DeepL API for Trados Studio providing different output than DeepL online

Question

Our company has a DeepL Pro subscription. We noticed a discrepancy between the translations provided by the free, online version of DeepL, and the ones provided by the DeepL API for Trados Studio 2021. 
 We assumed the API provided the same translations as the online version, but after running many tests in various language combinations, it&rsquo;s safe to say that this is not the case. 
 The online version is always noticeably better than the one provided by the API, but obviously we&rsquo;d rather use the API directly in the CAT tool. 
 Could you please let us know if this can be fixed in some way?

Jonathan Haerinck · Answer

Paul The fact that each segment is considered separately is a real shortcoming in the integration of NMT engines and LLM's in my opinion. You want to offer NMT engine integration in Trados Studio, but in fact, it's only a half-baked integration. How can you expect an engine to deliver quality output if it cannot consider the full context, and thus the relationship of a sentence to the sentences before and after it. It's like reading a book by reading 1 sentence every week. The NMT plugin translations are sometimes not even half as good as the translations in the web versions (whether NMT or LLM). So as a result, translators are now forced to shuffle between two systems (CAT tool for using the TMs and termbases), LLM/NMT web apps for good machine translation. It would be nice if translators could leverage all these resources (TM's, termbases, full NMT and LLM capability translation) in one single environment. I do fear that this functionality will become necessary if CAT tools want to compete with the emerging LLM translations and survive as a translation tool. I hope you will effectively consider this feedback in your product development and I really am looking forward to your proposed solutions (batch translating by sending larger numbers of segments in one go (configurable) and interactive translating that could offer the alternative translations-.

Jonathan Haerinck · Answer

Offcourse we did, we created the engines by importing a tmx-export file from our TM's.

Paul · Answer

Good point.. I think worth clarifying how " I think " this works in practice. ModernMT does not translate the entire document in one go, it also works segment by segment in a CAT tool as every MT provider does. But It says that it processes the translations segment by segment while leveraging a context vector generated by analysing the whole document upfront. I think this context vector provides broader insights into the entire file, allowing each segment to be translated with an understanding of the document&rsquo;s overall content. So not the same as actually translating the entire file in one go to ensure better translations in context (as you can do in probably any web translation tool... theory anyway!). 
 But it does suggest a clever way of handling this, so if you&rsquo;re working with multiple documents, such as 20 files, ModernMT handles each document individually, translating one segment at a time as requested by the CAT tool and returning translations immediately. While it operates in real-time, building its understanding dynamically, the context vector ensures document-level consistency without the need to process or translate the entire file in a single pass. 
 I like the theory, and I can only say " I think " this is what happens. But certainly an interesting approach to improving the quality. I think if we can apply this for all MT plugins that we support anyway: 
 
 batch translating by sending larger numbers of segments in one go (configurable) 
 interactive translating that could offer the alternative translations 
 
 Then we get closer to providing the improved quality wrt context users are looking for. 
 In the meantime I guess another approach would be to simply use document level translation for your files; align them; save to a TM; then work with that TM instead. 
 Jonathan Haerinck said: I don't think such an adjustment would have a big impact on translation costs, at least for NMT. In the DeeplPro subscription, the price is independent of translation volume; you can translate unlimited text. 
 Indeed you're right. It might be an issue for DeepL however as they base their rates on fair use so we will certainly have to be smart in how we implement this. In batch translation we would most likely not have to send unnecessary numbers of words... but interactively this could become a different matter. Even the use of lookahead has provoked discussion around this topic due to the amount of calls hitting their servers. 
 Nothing is ever as simple as we all might think it is ;-)

Paul · Answer

Jonathan Haerinck 
 I just tested this as well and may be reaching some sort of conclusion here: 
 If I send just one sentence then DeepL provides me with the very same result as you have in Trados Studio. The result you show in the web translator is the result of sending more segments at once... so more context. 
 If I also send all this in one go then I get this too: 
 
 So... this may be the main difference between the way this works. In the web you send everything in one go... the CAT tool doesn't do that. So it seems to me that we would need to work on an enhancement that could do something like this perhaps... options for: 
 
 batch translating by sending larger numbers of segments in one go (configurable) 
 interactive translating that could offer the alternative translations 
 
 I don't know yet how much of this is possible using the API. I can see full document translation is possible so we could convert to a text file for example and send a full document or several documents as needed... or something like that. But we can investigate this and if possible schedule something in for a future release.

Jonathan Haerinck · Answer

Paul I deliberately did not use glossaries in both environments so that the parameters are the same in both. As you can see in the 'hijab' example above, I also tested all examples with the 'Send source as plain text' option (as you can see above in the screenshot of the 'hijab' example). I could send you the Excel file with my examples. You can find another example here below (same settings: DeeplPro API versus DeeplPro web, no glossaries in either of them, Send source as plain text).

Paul · Answer

Jonathan Haerinck 
 Jonathan Haerinck said: I deliberately did not use glossaries in both environments so that the parameters are the same in both. As you can see in the 'hijab' example above, I also tested all examples with the 'Send source as plain text' option (as you can see above in the screenshot of the 'hijab' example). I could send you the Excel file with my examples. You can find another example here below (same settings: DeeplPro API versus DeeplPro web, no glossaries in either of them, Send source as plain text). 
 As I noted above... the problem is related to on the web you are not working one segment at a time.

Jonathan Haerinck · Answer

Paul Indeed, this was my suspicion since the very beginning of my tests. Surely this is a serious shortcoming for using NMT in Studio, it seems to me. The Web app of Deepl and LLMs like ChatGPT do interpret a piece of text in its entirety and thus give much better results. So it does seem appropriate to refine this further so that translators don't have to use both apps in parallel, as I have to do now (the TMs in Studio for the matches and fuzzy matches and Deepl Web for the pieces of NMT, whose output I then afterwards copy and paste into the Editor screen). Paulo Fernandes Here above you have proof of your suspicions. Deepl Web indeed translates better because it interprets the pieces of text as a whole, rather than each segment by itself.

Jonathan Haerinck · Answer

Paul I don't think such an adjustment would have a big impact on translation costs, at least for NMT. In the DeeplPro subscription, the price is independent of translation volume; you can translate unlimited text. 
 Probably this is a different issue for LLM translation. I wonder if the new AI assistant in Trados 2024 will also work this way. Soon we will get an Azure Open AI key to test this. If each segment is sent to the LLM separately, the output will probably not be as good as if the whole text or larger pieces of text were sent to the LLM. Eventually you will have the same problem: you will have to copy your text back from your editor, paste it into e.g. chat GPT, define your prompt and then paste the desired output from Chat GPT back into your Editor window.

Jonathan Haerinck · Answer

At least that's what they say on their website. Before Deepl, we used ModernMT for about 8 months. The problem with ModernMT is that, at least for the combination Dutch > French, the default output is really bad. For months I compared the output of the ModernMT plugin to that of the Deepl Web app, and in 99.99% of the cases Deepl performed better. Note that I am speaking only for the combination Dutch > French. Modern MT is adaptive, unlike Deepl, but even after correcting the same word thousands of times, the Modern MT engine keeps making the same error. I also did not really feel that the ModernMT engine took context into account, e.g., a demonstrative pronoun got a feminine inflection while the sentence before it was a masculine noun (with masculine article).

Paul · Answer

Valentina Cristofori 
 Examples?

Paul · Answer

Valentina Cristofori 
 No. 
 If "The online version is always noticeably better than the one provided by the API" then it should be simple to create some text you can share without compromising anyone's NDA. All you need to do is create a few sentences and then show us the online translation compared to the one you receive via the API.

Paul · Answer

Paulo Fernandes 
 Thank you... definitely not a simple case of all translations being different. But this is a good example we can take to DeepL for discussion as it is interesting to see this difference. We will be asked questions though that I tested here. 
 For example... if I use the basic free web DeepL I get this : 
 Der Ursprung des Serratus Posterior Inferior (SPI) ist durch eine d&uuml;nne, breite Aponeurose gekennzeichnet, die mit dem supraspin&ouml;sen Band der Wirbels&auml;ule und den Dornforts&auml;tzen der Wirbel T11-L2 verbunden ist. 
 If I use the plugin I get this: 
 Der Ursprung des Serratus Posterior Inferior (SPI) ist durch eine d&uuml;nne, breite Aponeurose gekennzeichnet, die mit dem supraspin&ouml;sen Band der Wirbels&auml;ule und den Dornforts&auml;tzen der Wirbel T11-L2 verbunden ist. 
 Same as your poor results so it seems as if the API is not using the Pro API at all. 
 We cannot test this using the web as we just get an API key from Deepl since we developed the plugin. We don't have our own account. But this at least gives us something to go on and we can investigate this with them to find out where the problem lies. 
 I can only think the specialised nature of your texts is why you found this because we would surely have seen more complaints by now otherwise.

Jonathan Haerinck · Answer

Hello Valentina Cristofori Paulo Fernandes and Paul Hello, We have been using the Deepl plugin for several months now and we also notice that the output of the plugin is remarkably worse than the output of the online version of Deepl. We translate from Dutch (Belgium) to French (Belgium). We are using a paid Deepl-subscription. Anyway, paid or not paid, this isn't the point. I did the test for both and the output in Deepl Web is the same for the paid as the free Web application. 
 I have done dozens of tests by translating the same input each time both in Trados and in the Deepl Web application and the differences are very remarkable. Unlike Paulo's example, ours are simple, non-technical texts. I have listed a few examples below. Anyway, there are always some differences between the output in the plugin and the web application (e.g. use of synonyms, infinitive instead of imperative, etc.) but I indicated the really heavy interpretation errors below in red in the second column (plugin output), and the correct or at least better version (Deepl Web), in green in the third column. First column is the source in Dutch. It even goes so far that we actually translate some pieces in Deepl Web and then copy and paste into our Editor screen of the project, which is very time consuming, just because sometimes the output is so bad in the plugin and needs too heavy postediting.

Paul · Answer

Valentina Cristofori Paulo Fernandes Jonathan Haerinck 
 Have you taken this up with DeepL? I would be interested in their take on this. Have you also played around with the options to send only plain text so that tags don't come into play at all? Have you tested the formalities to see how this effects things? 
 If you have, and if the problem is purely that the API is returning different translations then this is definitely something that will need to be addressed with DeepL as we have no control over that part. We just use the end points they provide to send the text and get a response back. 
 I have spent a little time trying to test this today, logged in with DeepL Pro and cannot get your preferred translations at all: 
 
 Perhaps you can give me some very clear and simple instructions on how to reproduce what you get? For me, testing the examples you gave (and it would be helpful if you provided text and not screenshots), the results look very much the same in the web application and the Trados via the API. 
 Maybe one thing work checking... do you have glossaries set up in the web that are not available to you in Trados as you have not added them? These are not the same.

RWS AppStore > 2. RWS AppStore Applications

DeepL API for Trados Studio providing different output than DeepL online

Top Replies