DeepL API for Trados Studio providing different output than DeepL online

Our company has a DeepL Pro subscription. We noticed a discrepancy between the translations provided by the free, online version of DeepL, and the ones provided by the DeepL API for Trados Studio 2021.

We assumed the API provided the same translations as the online version, but after running many tests in various language combinations, it’s safe to say that this is not the case.

The online version is always noticeably better than the one provided by the API, but obviously we’d rather use the API directly in the CAT tool.

Could you please let us know if this can be fixed in some way?

emoji
Parents
  • Hello   and  


    Hello,
    We have been using the Deepl plugin for several months now and we also notice that the output of the plugin is remarkably worse than the output of the online version of Deepl. We translate from Dutch (Belgium) to French (Belgium). We are using a paid Deepl-subscription. Anyway, paid or not paid, this isn't the point. I did the test for both and the output in Deepl Web is the same for the paid as the free Web application. 

    I have done dozens of tests by translating the same input each time both in Trados and in the Deepl Web application and the differences are very remarkable. Unlike Paulo's example, ours are simple, non-technical texts. I have listed a few examples below. Anyway, there are always some differences between the output in the plugin and the web application (e.g. use of synonyms, infinitive instead of imperative, etc.) but I indicated the really heavy interpretation errors below in red in the second column (plugin output), and the correct or at least better version (Deepl Web), in green in the third column. First column is the source in Dutch. It even goes so far that we actually translate some pieces in Deepl Web and then copy and paste into our Editor screen of the project, which is very time consuming, just because sometimes the output is so bad in the plugin and needs too heavy postediting.

    Screenshot of a translation comparison table with three columns. The first column contains Dutch text, the second column shows incorrect French translations highlighted in red, and the third column shows correct French translations in green.

    Close-up of a translation comparison table highlighting specific translation errors. Incorrect French translations are marked in red, and the correct or preferred translations are indicated in green.

    Screenshot of a translation comparison table with bullet points. The first column is in Dutch, the second column shows French translations with errors in red, and the third column has the correct translations in green.

    Close-up of a translation comparison table showing a list of Dutch sentences and their French translations. Errors in the plugin output are highlighted in red, while the correct web application translations are in green.
    Screenshot of a detailed translation comparison table with extensive text. The first column is in Dutch, the second column shows French translations with errors in red, and the third column has the correct translations in green.
    Close-up of a translation comparison table with longer paragraphs. The first column is in Dutch, the second column shows French translations with errors in red, and the third column has the correct translations in green.

    emoji


    Generated Image Alt-Text
    [edited by: RWS Community AI at 6:33 AM (GMT 1) on 26 Sep 2024]
  • Thank you for your comment. In our tests, it seemed to us that formatting plays a major role in mistranslations. Were your texts formatted in any way (bold, italic, with tags, etc.)? We thought that perhaps the browser version performed better because there was no formatting, whereas in Trados it looked as if the segments were translated in chunks, broken up whenever there were tags or formatting changes.  

    emoji
  •  

    I guess we'll see how much users like it when they are faced with the additional costs of sending content for translation.  As a batch task it should be fine... but for interactive translation it may not be and definitely has to be configurable.

    Certainly this has isn't the first time this has come up, in the context of AI and also NMT providers.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

    emoji
  •  I don't think such an adjustment would have a big impact on translation costs, at least for NMT. In the DeeplPro subscription, the price is independent of translation volume; you can translate unlimited text.

    Probably this is a different issue for LLM translation. I wonder if the new AI assistant in Trados 2024 will also work this way. Soon we will get an Azure Open AI key to test this. If each segment is sent to the LLM separately, the output will probably not be as good as if the whole text or larger pieces of text were sent to the LLM. Eventually you will have the same problem: you will have to copy your text back from your editor, paste it into e.g. chat GPT, define your prompt and then paste the desired output from Chat GPT back into your Editor window.

    emoji
  • Good point! From my experience there is also a big difference in quality between a large text translated on free web and smaller portions of the same text resulting in a better quality for smaller junks of text (but this is a web with web comparison).

    emoji
  • I wonder if the new AI assistant in Trados 2024 will also work this way.

    Of course.  Every MT Provider in Trados works this way.  You can improve it yourself somewhat with all MT providers including DeepL... if the document is formatted with paragraphs, by using paragraph segmentation.  This way you will be sending a complete paragraph each time you send the request.  The downside of course being that all your other resources are most likely sentence based so you won't get any benefit from them; and you'll be storing paragraphs in each Translation Unit in your TM as opposed to sentences.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

    emoji
  •      

    Of course.  Every MT Provider in Trados works this way. 

    Not really. ModernMT, for example, reads the entire document to get the context and uses this to enhance the quality of the translation.

    Let me quote their explanation:

    ModernMT can compute a context-vector by automatically analyzing the whole source document right from the beginning. The context vector indeed represents the context of the whole document and this information is passed in every single Translate API call and used to translate every sentence but having a broader context. This Context Vector feature is automatically handled by the plugin.

    Upon document opening, our plugin automatically starts an analysis of the entire document to be translated. The result of the analysis (the context vector) goes from one segment to another. This means that, even though the translation is carried out segment by segment, the engine takes into consideration the context (the context vector) at each single segment.


    Apparently, this is something that Deepl (or at least the plugin) does not do.

    emoji
  •  
    At least that's what they say on their website. Before Deepl, we used ModernMT for about 8 months. The problem with ModernMT is that, at least for the combination Dutch > French, the default output is really bad. For months I compared the output of the ModernMT plugin to that of the Deepl Web app, and in 99.99% of the cases Deepl performed better. Note that I am speaking only for the combination Dutch > French. Modern MT is adaptive, unlike Deepl, but even after correcting the same word thousands of times, the Modern MT engine keeps making the same error. I also did not really feel that the ModernMT engine took context into account, e.g., a demonstrative pronoun got a feminine inflection while the sentence before it was a masculine noun (with masculine article).

    emoji
  • Modern MT is adaptive, unlike Deepl, but even after correcting the same word thousands of times, the Modern MT engine keeps making the same error.

    HI  

    Did you attach a TM in the ModernMT engine?

    emoji
  •   Offcourse we did, we created the engines by importing a tmx-export file from our TM's.

    emoji
  •  

    Good point.. I think worth clarifying how "I think" this works in practice.  ModernMT does not translate the entire document in one go, it also works segment by segment in a CAT tool as every MT provider does.  But It says that it processes the translations segment by segment while leveraging a context vector generated by analysing the whole document upfront.  I think this context vector provides broader insights into the entire file, allowing each segment to be translated with an understanding of the document’s overall content.  So not the same as actually translating the entire file in one go to ensure better translations in context (as you can do in probably any web translation tool... theory anyway!).

    But it does suggest a clever way of handling this, so if you’re working with multiple documents, such as 20 files, ModernMT handles each document individually, translating one segment at a time as requested by the CAT tool and returning translations immediately.  While it operates in real-time, building its understanding dynamically, the context vector ensures document-level consistency without the need to process or translate the entire file in a single pass.

    I like the theory, and I can only say "I think" this is what happens.  But certainly an interesting approach to improving the quality.  I think if we can apply this for all MT plugins that we support anyway:

    • batch translating by sending larger numbers of segments in one go (configurable)
    • interactive translating that could offer the alternative translations

    Then we get closer to providing the improved quality wrt context users are looking for. 

    In the meantime I guess another approach would be to simply use document level translation for your files; align them; save to a TM; then work with that TM instead.

    I don't think such an adjustment would have a big impact on translation costs, at least for NMT. In the DeeplPro subscription, the price is independent of translation volume; you can translate unlimited text.

    Indeed you're right.  It might be an issue for DeepL however as they base their rates on fair use so we will certainly have to be smart in how we implement this.  In batch translation we would most likely not have to send unnecessary numbers of words... but interactively this could become a different matter.  Even the use of lookahead has provoked discussion around this topic due to the amount of calls hitting their servers.

    Nothing is ever as simple as we all might think it is ;-)

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

    emoji
  •  
    The fact that each segment is considered separately is a real shortcoming in the integration of NMT engines and LLM's in my opinion. You want to offer NMT engine integration in Trados Studio, but in fact, it's only a half-baked integration. How can you expect an engine to deliver quality output if it cannot consider the full context, and thus the relationship of a sentence to the sentences before and after it. It's like reading a book by reading 1 sentence every week. The NMT plugin translations are sometimes not even half as good as the translations in the web versions (whether NMT or LLM). So as a result, translators are now forced to shuffle between two systems (CAT tool for using the TMs and termbases), LLM/NMT web apps for good machine translation. It would be nice if translators could leverage all these resources (TM's, termbases, full NMT and LLM capability translation) in one single environment. I do fear that this functionality will become necessary if CAT tools want to compete with the emerging LLM translations and survive as a translation tool. I hope you will effectively consider this feedback in your product development and I really am looking forward to your proposed solutions (batch translating by sending larger numbers of segments in one go (configurable) and interactive translating that could offer the alternative translations-.


    emoji
  • Reply
    •  
      The fact that each segment is considered separately is a real shortcoming in the integration of NMT engines and LLM's in my opinion. You want to offer NMT engine integration in Trados Studio, but in fact, it's only a half-baked integration. How can you expect an engine to deliver quality output if it cannot consider the full context, and thus the relationship of a sentence to the sentences before and after it. It's like reading a book by reading 1 sentence every week. The NMT plugin translations are sometimes not even half as good as the translations in the web versions (whether NMT or LLM). So as a result, translators are now forced to shuffle between two systems (CAT tool for using the TMs and termbases), LLM/NMT web apps for good machine translation. It would be nice if translators could leverage all these resources (TM's, termbases, full NMT and LLM capability translation) in one single environment. I do fear that this functionality will become necessary if CAT tools want to compete with the emerging LLM translations and survive as a translation tool. I hope you will effectively consider this feedback in your product development and I really am looking forward to your proposed solutions (batch translating by sending larger numbers of segments in one go (configurable) and interactive translating that could offer the alternative translations-.


      emoji
    Children
    No Data