DeepL API for Trados Studio providing different output than DeepL online

Our company has a DeepL Pro subscription. We noticed a discrepancy between the translations provided by the free, online version of DeepL, and the ones provided by the DeepL API for Trados Studio 2021.

We assumed the API provided the same translations as the online version, but after running many tests in various language combinations, it’s safe to say that this is not the case.

The online version is always noticeably better than the one provided by the API, but obviously we’d rather use the API directly in the CAT tool.

Could you please let us know if this can be fixed in some way?

emoji
Parents
  • Hello   and  


    Hello,
    We have been using the Deepl plugin for several months now and we also notice that the output of the plugin is remarkably worse than the output of the online version of Deepl. We translate from Dutch (Belgium) to French (Belgium). We are using a paid Deepl-subscription. Anyway, paid or not paid, this isn't the point. I did the test for both and the output in Deepl Web is the same for the paid as the free Web application. 

    I have done dozens of tests by translating the same input each time both in Trados and in the Deepl Web application and the differences are very remarkable. Unlike Paulo's example, ours are simple, non-technical texts. I have listed a few examples below. Anyway, there are always some differences between the output in the plugin and the web application (e.g. use of synonyms, infinitive instead of imperative, etc.) but I indicated the really heavy interpretation errors below in red in the second column (plugin output), and the correct or at least better version (Deepl Web), in green in the third column. First column is the source in Dutch. It even goes so far that we actually translate some pieces in Deepl Web and then copy and paste into our Editor screen of the project, which is very time consuming, just because sometimes the output is so bad in the plugin and needs too heavy postediting.

    Screenshot of a translation comparison table with three columns. The first column contains Dutch text, the second column shows incorrect French translations highlighted in red, and the third column shows correct French translations in green.

    Close-up of a translation comparison table highlighting specific translation errors. Incorrect French translations are marked in red, and the correct or preferred translations are indicated in green.

    Screenshot of a translation comparison table with bullet points. The first column is in Dutch, the second column shows French translations with errors in red, and the third column has the correct translations in green.

    Close-up of a translation comparison table showing a list of Dutch sentences and their French translations. Errors in the plugin output are highlighted in red, while the correct web application translations are in green.
    Screenshot of a detailed translation comparison table with extensive text. The first column is in Dutch, the second column shows French translations with errors in red, and the third column has the correct translations in green.
    Close-up of a translation comparison table with longer paragraphs. The first column is in Dutch, the second column shows French translations with errors in red, and the third column has the correct translations in green.

    emoji


    Generated Image Alt-Text
    [edited by: RWS Community AI at 6:33 AM (GMT 1) on 26 Sep 2024]
  • Thank you for your comment. In our tests, it seemed to us that formatting plays a major role in mistranslations. Were your texts formatted in any way (bold, italic, with tags, etc.)? We thought that perhaps the browser version performed better because there was no formatting, whereas in Trados it looked as if the segments were translated in chunks, broken up whenever there were tags or formatting changes.  

    emoji
  •  

    I deliberately did not use glossaries in both environments so that the parameters are the same in both. As you can see in the 'hijab' example above, I also tested all examples with the 'Send source as plain text' option (as you can see above in the screenshot of the 'hijab' example). I could send you the Excel file with my examples. You can find another example here below (same settings: DeeplPro API versus DeeplPro web, no glossaries in either of them, Send source as plain text).

    As I noted above... the problem is related to on the web you are not working one segment at a time.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

    emoji
  •  Indeed, this was my suspicion since the very beginning of my tests. Surely this is a serious shortcoming for using NMT in Studio, it seems to me. The Web app of Deepl and LLMs like ChatGPT do interpret a piece of text in its entirety and thus give much better results. So it does seem appropriate to refine this further so that translators don't have to use both apps in parallel, as I have to do now (the TMs in Studio for the matches and fuzzy matches and Deepl Web for the pieces of NMT, whose output I then afterwards copy and paste into the Editor screen).

      Here above you have proof of your suspicions. Deepl Web indeed translates better because it interprets the pieces of text as a whole, rather than each segment by itself.

    emoji
  •  

    I guess we'll see how much users like it when they are faced with the additional costs of sending content for translation.  As a batch task it should be fine... but for interactive translation it may not be and definitely has to be configurable.

    Certainly this has isn't the first time this has come up, in the context of AI and also NMT providers.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

    emoji
  •  I don't think such an adjustment would have a big impact on translation costs, at least for NMT. In the DeeplPro subscription, the price is independent of translation volume; you can translate unlimited text.

    Probably this is a different issue for LLM translation. I wonder if the new AI assistant in Trados 2024 will also work this way. Soon we will get an Azure Open AI key to test this. If each segment is sent to the LLM separately, the output will probably not be as good as if the whole text or larger pieces of text were sent to the LLM. Eventually you will have the same problem: you will have to copy your text back from your editor, paste it into e.g. chat GPT, define your prompt and then paste the desired output from Chat GPT back into your Editor window.

    emoji
  • Good point! From my experience there is also a big difference in quality between a large text translated on free web and smaller portions of the same text resulting in a better quality for smaller junks of text (but this is a web with web comparison).

    emoji
  • I wonder if the new AI assistant in Trados 2024 will also work this way.

    Of course.  Every MT Provider in Trados works this way.  You can improve it yourself somewhat with all MT providers including DeepL... if the document is formatted with paragraphs, by using paragraph segmentation.  This way you will be sending a complete paragraph each time you send the request.  The downside of course being that all your other resources are most likely sentence based so you won't get any benefit from them; and you'll be storing paragraphs in each Translation Unit in your TM as opposed to sentences.

    Paul Filkin | RWS Group

    ________________________
    Design your own training!

    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

    emoji
  •      

    Of course.  Every MT Provider in Trados works this way. 

    Not really. ModernMT, for example, reads the entire document to get the context and uses this to enhance the quality of the translation.

    Let me quote their explanation:

    ModernMT can compute a context-vector by automatically analyzing the whole source document right from the beginning. The context vector indeed represents the context of the whole document and this information is passed in every single Translate API call and used to translate every sentence but having a broader context. This Context Vector feature is automatically handled by the plugin.

    Upon document opening, our plugin automatically starts an analysis of the entire document to be translated. The result of the analysis (the context vector) goes from one segment to another. This means that, even though the translation is carried out segment by segment, the engine takes into consideration the context (the context vector) at each single segment.


    Apparently, this is something that Deepl (or at least the plugin) does not do.

    emoji
  •  
    At least that's what they say on their website. Before Deepl, we used ModernMT for about 8 months. The problem with ModernMT is that, at least for the combination Dutch > French, the default output is really bad. For months I compared the output of the ModernMT plugin to that of the Deepl Web app, and in 99.99% of the cases Deepl performed better. Note that I am speaking only for the combination Dutch > French. Modern MT is adaptive, unlike Deepl, but even after correcting the same word thousands of times, the Modern MT engine keeps making the same error. I also did not really feel that the ModernMT engine took context into account, e.g., a demonstrative pronoun got a feminine inflection while the sentence before it was a masculine noun (with masculine article).

    emoji
  • Modern MT is adaptive, unlike Deepl, but even after correcting the same word thousands of times, the Modern MT engine keeps making the same error.

    HI  

    Did you attach a TM in the ModernMT engine?

    emoji
  •   Offcourse we did, we created the engines by importing a tmx-export file from our TM's.

    emoji
  • Reply Children
    No Data