Marta Bartnicka

I wonder what the basis is you bill your clients on - or they bill you on. If I bring my bike to the workshop, I pay for it to be fixed. If the mechanic tries one fix and it does not work, so that he has to try another fix, that is his business. If he is very quick and efficient, the deal will be better for him... That is what I think about your question, too. If I work on a segment several times, that is because I did not get it quite right the first time - my bad.

If you want to be paid for your effort, not for your results (or if you want to pay your post-MT-translator accordingly), you'd pay an hourly rate...

Or you use Post-Edit Compare and create several versions: First pass, create new version, second pass etc. Then you bill for both passes individually.

Or maybe I am misunderstanding your question.

Daniel

Hello Daniel!

To keep your bike metaphor: it's not about pricing the fixing of a bike, it's about selecting the best wrench 

My clients need to calculate PEM not for paying the translators for their effort (I would not recommend that model either), but to choose the best MT for a given language pair and domain. I usually advise to combine 3 factors: PE time, PEM/edit distance and translators' subjective opinion (sometimes also proofreaders' opinion). I wouldn't say it's "my bad" when a translator changes segments >1 time - it is more about the style of work, e.g. getting the meaning right in the first pass and cleaning up the terminology/consistency (usual generic MT flaws) in the second pass. So, I am looking for a method to get the total PEM between MT proposal and the final translator's output, no matter how many times it was edited...?

Post-Edit Compare would be an option but the same client observes an issue similar to this and cannot use PEC in production: community.sdl.com/.../88494

Hello Daniel!

To keep your bike metaphor: it's not about pricing the fixing of a bike, it's about selecting the best wrench 

My clients need to calculate PEM not for paying the translators for their effort (I would not recommend that model either), but to choose the best MT for a given language pair and domain. I usually advise to combine 3 factors: PE time, PEM/edit distance and translators' subjective opinion (sometimes also proofreaders' opinion). I wouldn't say it's "my bad" when a translator changes segments >1 time - it is more about the style of work, e.g. getting the meaning right in the first pass and cleaning up the terminology/consistency (usual generic MT flaws) in the second pass. So, I am looking for a method to get the total PEM between MT proposal and the final translator's output, no matter how many times it was edited...?

Post-Edit Compare would be an option but the same client observes an issue similar to this and cannot use PEC in production: community.sdl.com/.../88494

Marta Bartnicka

Marta Bartnicka said:

So, I am looking for a method to get the total PEM between MT proposal and the final translator's output, no matter how many times it was edited...?

If you ask the translator to unstall Qualitivity and get them to agree to share the reports generated, then this can record the number of times they edit a segment, the keystrokes used, when they did it, the time spent in each operation etc.  Basically it can record everything the translator does to that file.

But how you would use this information to pay the translator is another question altogether.  It does feel a little over complicated.  If you want this information to help provide data for post-translation analysis based on domain and effort for the engine used so you eventually have data to help select the right engine then this "may" be useful.  But in many ways I think it's more of an academic exercise and not something you can really use productively in an operational environment as the results are too late.

MT Comparison is quite useful for this sort of thing because it helps you make a judgement based on different engines for a sample of the data in the project and then hopefully achieve the best solution in a production environment: https://appstore.sdl.com/language/app/mt-comparison/933/

Certainly an interesting and relevant discussion.

Hello Paul!

Thank you for the considerations. Paying translators per post-editing effort is certainly not a good idea and not my question either, so let's leavethat be (or rather, not be).

But selecting a MT engine is a real-life case. While not all translators may have Qualitivity installed and activated all the time, it does make sense to use it for a limited period of time and on selected "typical" projects, if more than one MT is available for a language pair and domain. It is not too late but just in time to evaluate the first results (3 factors as I wrote above) and decide about further production use.

I know to what level of detail Qualitivity can record data, and I am thinking how to make the best use of it. While post-editing time can be calculated in an easy and reliable way, the PEM has to be "handled with care" as I see it. Btw. in academic research I spotted recently, they just drop all segments visited more than once I am not sure I'd recommend that to my clients, though.

The more I think about it, the more I feel like reaching to a fellow programmer for writing a nice little tool to calculate Levenshtein distance between raw MT proposals and final translators output!

Marta Bartnicka said:

The more I think about it, the more I feel like reaching to a fellow programmer for writing a nice little tool to calculate Levenshtein distance between raw MT proposals and final translators output!

Why not use Post-Edit Compare?  You save the raw MT as your baseline and can then compare any version of the project you save after that with it.  If you make sure that your client who gets the error you mentioned is fully upto date wth the version of 2017 or 2019 they are using then we'll be happy to take a look at any project causing this problem.

Would be nice :) I will ask the client if they want to debug the Post-Edit Compare issue.

Btw. the same error messages had been reported in this thread and sample files were provided by Pavel Brunda - I would be interested in the solution of that case: community.sdl.com/.../post-edit-compare-error

In fact I don't think there was a resolution for that one... I think I missed it and the files have expired.  Frankly I'm inclined to believe it would have been a user specific thing non-related to the plugin itself as this is a fairly heavily used plugin, especially by SDL internally, and we don't see these sort of problems.

But I'll go back and check just in case.

Marta Bartnicka

Konstantin Savenkov has a different approach to a similar question. You might be aware of this already, but I found it useful: https://www.slideshare.net/KonstantinSavenkov/state-of-the-machine-translation-by-intento-stock-engines-jun-2019

Daniel

Hello Daniel! Yes, I know the Intento reports and their methodology. In general, Intento's evaluation of stock engines is a good starting point if looking for MT(s). Still, the time may come when we need to figure out which is more productive - e.g. a stock model with grand linguistic quality, or a trained model with perfected terminology

How to implement LEPOR on data gathered from Trados/Qualitivity, would be a separate question, though...