The SDL Machine Translation (MT) research team recently announced that our latest machine learning innovations and development strategies with Neural MT have resulted in a breakthrough that clearly demonstrates a significant and substantial leap forward. When testing our Russian to English MT system, the output, when measured against an extensive suite of comparative experiments to verify and validate the outstanding results, outperformed all industry standards, setting a benchmark for Russian to English machine translation. Over 90% of the system’s output has been labeled as a perfect translation by professional Russian-English translators.

Those who have been momnitoring the progress of Neural MT systems may be aware that Russian to English has been a particularly challenging direction for MT developers.

Adolfo Hernandez, CEO, SDL

It was the Russian language that first inspired the science and research behind machine translation,” said Adolfo Hernandez, CEO, SDL. “Since then it has always been a major challenge for the community. SDL has deployed breakthrough research strategies to master these difficult languages, and support the global expansion of its enterprise customers. We have pushed the boundaries and raised the performance bar even higher, and we are now paving the way for leadership in other complex languages.”

The linguistic properties and intricacies of the Russian language relative to English make it particularly challenging for MT systems to model. Russian is a highly inflected language with different syntax, grammar, and word order compared to English. Given the complexities created by these differences between the Russian and English language, raising the translation quality has been an ongoing focus of the SDL Machine Learning R&D team.

SDL Neural MT Russian to English results

Much of the enthusiasm for Neural MT is driven by the degree of fluency and naturalness of the output, and its ability to produce a large number of sentences that look like they are very fluent and look like they are from the human tongue. We have seen that often the early results with Neural MT output show that it is considered to be clearly better to human evaluators, even though established MT evaluation metrics such as the BLEU score may only show nominal or no improvements.

The improvements to human assessment are most noticeable when considering fluency and word order issues with machine translation output. However, the most common automatic quality metric used by MT developers during the R&D phase is still the BLEU score, so it is important to incorporate human assessments into the scoring methodology.

The strategy adopted by the SDL researchers was to use professional human assessment as a primary means to assess the MT quality. We wanted to know the human-perceived translation quality of SDL Neural MT, and understand how it compared to the human-perceived translation quality of an actual human translation. SDL builds custom MT engines for production use on a regular basis and has developed an accurate and reliable evaluation methodology to assess the quality of MT output that minimizes human bias.

A team of professional human Russian to English translators were shown a set of blind and random translations that came from any of the following systems and were not identified in any way:

A human translation of the test set by a professional translator
State-of-the-art Rule-Based MT output
State-of-the-art Statistical MT output
SDL Neural MT output

The evaluation team was shown the source Russian sentence and the blind translation output and was asked to score each output on a scale that had “perfect translation” on one end, and “completely wrong” on the other end. For each system, the percentage of translations that were scored as high as the human translation score was computed. The final results are listed below.

The SDL research shows that its Neural MT system outperformed all industry standards, setting a benchmark for Russian to English machine translation, with 95% of the system’s output labeled as equivalent to a human translation in terms of quality by professional Russian-English translators.

Left to Right: Amos Kariuki, Ling Tsou, Dragos Munteanu, Samad Echihabi, Quinn Lam, Wes Feely, and William Tambellini

With over fifteen years of research and innovation in machine translation, our scientists and engineers took up the challenge to bring Neural MT to the next level,” said Samad Echihabi, Head of Machine Learning R&D, SDL. “We have been evolving, optimizing and adapting our neural technology to deal with highly complex translation tasks such as Russian to English, with phenomenal results. A machine running SDL’s Neural MT technology can now produce translations of Russian text virtually indistinguishable from what Russian-English bilingual humans can produce.”

SDL latest Neural MT technology is optimized for both accuracy and fluency and provides a powerful paradigm to deal with morphologically rich and complex languages. While the focus of the SDL tests and measurements was the Russian to English system, the strategies deployed by the SDL team are expected to be compatible with and be of benefit to other complex and morphologically rich languages.

It is interesting to note that the best Russian-English SMT systems, even after 10+ years of research, were only marginally better than the best Russian-English RBMT systems, if at all. This points to the significant challenge presented by the Russian to English language combination and explains why RBMT systems have been preferred by many industrial users until quite recently. The new SDL Neural MT system is very likely to accelerate this transition.

Why is Russian difficult for MT?

Russian has always been considered to be one of the most difficult languages in MT, mostly because it is very different linguistically from English. Russian differs from English significantly in inflection, morphology, word order and gender associations with nouns.

Inflection

Unlike English, Russian is a highly inflected language. Suffixes on nouns mark 6 distinct cases, which determine the role of the noun in the sentence (whether it’s the subject, the direct object, the indirect object, something being possessed, something used as an instrument, or the object of a preposition). For example, all of these are different forms of the word “book.”

That’s 12 forms of the same word, which are used depending on what role the word is playing in the sentence. “But they’re not all distinct; you can have the same form for different roles, like the singular genitive & the plural nominative,” says Wes Feely, Senior Computational Linguist, SDL.

Additionally, like Spanish or French every noun has a gender. The word for “book” is feminine, but this is an arbitrary categorization. “There’s no reason why a book (книга kníga) is feminine and why a table (стол stól) is masculine,” explains Wes. “But it matters because the case suffixes are different for each gender (masculine, feminine, or neuter). So while there are 12 different forms of the word “book” and 12 different forms of the word “table”, they don’t share the same set of suffixes. When adjectives modify nouns, they need to agree with the noun, taking the same (or similar) suffix.”

Also, like Spanish or French, verbs conjugate depending on the tense (past vs. non-past), person (I vs. you vs. he/she/it), number (singular vs. plural), etc. So one verb may have several different forms, as well.

Word order

In English, we use word order to accomplish the same thing as the suffixes on nouns in Russian. Because Russian has these case markings, their word order is much freer. For example, these are all acceptable ways of saying “I went to the shop.”

Sample Output from SDL Russian to English Neural MT system

Essentially, all orderings are possible, except that the preposition “to” (в v) must precede the word for “shop” (магазин magazin). You can imagine that as sentences get longer, the number of possible sentence order structures increase. There are some limits on this: some orders in this example sound strange or archaic, and others are only used to emphasize where you’re going or who is going. But there are certainly more ways of saying the same thing than English, which is stricter in its word order.

SDL’s latest Neural MT technology is able to deal with all the Russian language challenges described above and can produce fluent and accurate translations. Below are some examples from the new SDL Neural MT Russian-English system.

Russian	До Уистлера, расположенного в провинции Британская Колумбия, можно быстро добраться от Ванкувера на автомобиле или самолете.
SDL Neural MT English Output	Whistler, located in British Columbia, is easily accessible from Vancouver by car or plane.
Human English Output	Whistler, British Columbia, is quickly accessible from Vancouver by road or air.

Russian	Фестивали, спа, рестораны и бары сочетаются с бесконечными возможностями досуга на свежем воздухе, делая Уистлер идеальным местом, где вы можете отдохнуть и расслабиться.
SDL Neural MT English Output	Festivals, spa, restaurants and bars combine with endless outdoor activities, making Whistler the ideal place to relax and unwind.
Human English Output	Festivals, spas, restaurants, and bars combine with endless outdoor activities to make Whistler the ultimate place to escape and unwind.

Russian	Директор оперативного управления Международного комитета Красного Креста Доминик Стиллхарт сообщил в воскресенье на пресс-конференции в Сане, что с 27 апреля по 13 мая в стране умерло от холеры 115 человек.
SDL Neural MT English Output	The Director of Operations of the International Committee of the Red Cross, Dominic Stillhart, reported on Sunday at a press conference in Sana’a that 115 people died from cholera from April 27 to May 13.
Human English Output	On Sunday, Dominik Stillhart, director of operations of the International Committee of the Red Cross, said during a press conference in Sana’a that 115 people died from cholera in the country between April 27 and May 13.

Russian	Рынок акций США в среду, вероятнее всего, начнет торговую сессию умеренным ростом на 0,4-0,5% в рамках коррекции к падению предыдущего дня, вызванному напряжением в торговых отношениях между США, Китаем и некоторыми другими странами.
SDL Neural MT English Output	The US stock market is likely to start its trading session on Wednesday 0.4-0.5% as part of an adjustment to the fall of the previous day caused by the tension in trade relations between the United States, China, and some other countries.
Human English Output	The US stock market on Wednesday, will most likely start a trade session with a moderate growth of 0.4-0.5% as part of the correction to the fall of the previous day caused by tension in trade relations between the USA, China, and some other countries.

It is important to qualify that these results only reflect the results of generic MT systems and of generic sentences as shown above. The SDL research noted that the generic Neural MT system did not perform as well on domain-specific data. As a supplier of MT solutions to the enterprise, SDL will typically adapt MT systems to the unique needs of each enterprise customer’s domain. This adaptation is also an especially challenging task with Neural MT models. In other experiments with domain adapted MT systems, the SDL research team noted that there were further improvements in perceived quality and they documented that, adaptation of the SDL Neural technology provided a 30% improvement over the generic neural engine on domain-specific data.

SDL Neural MT for the enterprise

The SDL Neural MT breakthrough accomplishment comes soon after several other announcements related to their ongoing research and development, and substantial progress in taking Neural MT from a research environment to a deployable enterprise-ready technology.

It is remarkable to see such a leap in translation quality with SDL’s latest neural technology. We are currently working on transitioning this advancement from our R&D lab to our enterprise customers,” said Quinn Lam, Senior Product Manager, SDL. “Planned for release this summer, the latest version of SDL Enterprise Translation Server (ETS) will be powered by this fully productized state-of-the-art Neural technology. Stay tuned!”

Another key requirement for successful MT adoption in the enterprise is the ability to get the system to learn and adapt to enterprise-specific linguistic requirements and preferences. This has been especially challenging with Neural MT technology which, until now, has been difficult to do without undermining the fluency and output quality. SDL researchers recently figured out how to augment Neural MT with dictionary capabilities. This means enterprises can easily adapt SDL Neural MT across multiple departments that have differing terminology, yet still maintain the translation fluency that this latest generation of MT technology is acclaimed for.

SDL’s latest dictionary feature sets a new industry standard for user control over automated translations, allowing users across the enterprise to use different dictionaries without impacting the quality of the translations. SDL ETS Dictionary capabilities include:

Controls that allow an enterprise to enforce multiple terminology and translation preferences for the same word, something that is necessary for different departments who may have unique interpretations for the same word or term.
Easy implementation of preferred terminology and personalization by any user with no upfront technical knowledge or training required.
Deployment of multiple dictionaries in a single engine at the same time, allowing multiple departments with differing needs to optimize the MT engine differently.
Terminology preferences that can be changed and modified on an ongoing basis to accommodate changing business and communication priorities.

The SDL research team has also recently found new ways to address a major issue with Neural MT systems, namely improving system speed and throughput. This has been a general problem for Neural MT system deployments as these deployments can be expensive and slow, and any ability to improve the efficiency of private deployments are especially valuable in the enterprise use context.

Left to Right: Gonzalo Iglesias, Bill Byrne, Eva Hasler & Adrià De Gispert

We now have has several other initiatives underway and will continue to introduce new features and capabilities emerging from their Neural MT research over the coming year as we bring innovative research ideas from the lab to the production deployment arena. As Samad added, “While there is great excitement about Neural MT, it is clear that as we explore further the science, we already see signs that we will continue to make progress and we look forward to bringing the most relevant innovations to market for our customers.”

Blog

Language Weaver Live 2023

The Global Data Explosion in the Legal Industry

Latest SDL Machine Translation Sets New Standard as Essential Business Technology Capable of Instantly Translating Any Type of Content

The Real Problems with Neural Machine Translation

A Detailed study of Google Translate Performance across 103 language cobinations

Is the Future of BLEU Getting Paler?

Language RPA: The Coming of Age for Language Technologies

Language RPA: The Coming of Age for Language Technologies

The Answer to Writer's Block?

Can Adaptable Machine Translation Drive Your Digital Transformation?

The Challenge of using Open Source MT

Understanding MT Quality: What Really Matters?

Understanding MT Quality - BLEU Scores

SDL Enterprise Translation Server (ETS) 8.3 General Availability

SDL Cracks Russian Neural Machine Translation -- SDL's Technology Leadership Profile

SDL Neural MT Russian to English results

Why is Russian difficult for MT?

SDL Neural MT for the enterprise