Setting up an MT Post-Edit solution does not stop with providing a MT engine. Once an engine is live, our clients want to know how it performs so that they can successfully manage costs, post-editing resources and future engine improvements.

The science of measuring and predicting MT productivity gains can be a little confusing at times and we would like to shed some light on a popular evaluation method, the Translation Edit Rate (TER).

So, what is TER? TER and its variants are a metrics to assess post-editing effort. Put simply, TER measures the number of edits that are necessary to go from the raw MT output to a final edited version. As such, it is a helpful metric to measure typing and editing effort. But let’s stop for a minute and take a closer look at what post-editing really entails. Is it only typing and editing? On closer examination, the post-editor carries out quite a number of tasks during the editing process: he or she reads the source and target, considers necessary edits, validates terminology through TM and MultiTerm searches, ensures consistency as well as text flow and guarantees that the final text meets client expectations. As we can see, post-editing really amounts to much more than purely typing and editing.

So, how does this correlate to TER? All the efforts made by the post-editor to deliver a high-quality end product are not really accounted for by simply measuring the final edits. TER calculates the minimum number of edits necessary to go from MT to final post-edited version, but we have to remember that post-editors are only human: post-editing is not necessarily a linear process but one where post-editors may rethink their edits or make changes. This cannot be captured by TER.

On the whole, TER is a very useful metric to assess editing effort and provide assistance to MT developers for improving MT performance. When it comes to measuring human effort and fair compensation for post-editors, we need to look further than the shortest editing distance between MT and finished product and recognize the value that human post-editors bring to the MT process through their work and feedback.