Translation memory management: duplicate TUs or overwritten TUs

RE: Translation memory management: duplicate TUs or overwritten TUs

Pascal Zotto — Fri, 02 Feb 2018 11:25:25 GMT

Hi Keenan,

The filter I mentionned was in combination with my idea, with the way Trados works not it certainly is not possible in your case.

Still, as a programmer and database admin I know there are better solutions to avoid your problem but I'm not sure if SDL could implement that one on TMs easily.

They would need to set up a second table with foreign keys on the TU table for TUs that exists for different manuals. In that table they link the fields connected to a TU. By doing this you can keep clean fields and determine several reference per TU. With this setup my suggested filter would still work and only show up one TM match with the required fields.

For those not knowing much about DB structure it would be like (for your example):

Table 1: (contains only TUs)

ID Source Target

TU#1 Source: パワーコンディショナー Target: PV inverter
TU#2 Source: パワーコンディショナー Target: power conditioner

Table 2: (contains the fields linked to TUs for various projects)

ID ID (foreign key from other table) Field
1 TU#1 Filename: P11 manual
2 TU#1 Filename: T11 manual
3 TU#1 Filename: R11 manual
4 TU#1 Filename: S11 manual
5 TU#1 Filename: Q11 manual
6 TU#2 Filename: F11 manual

with filters Studio would query this like this:

Filter for Filename: T11 manual would result in showing:

TU#1 Source: パワーコンディショナー Target: PV inverter Filename: T11 manual

As a user, programmer and database admin I understand that users always want their ideas implemented straight forward (I see it every day, when my colleagues ask me to implement (for them) some obvious idea quickly into the app and these ideas often make sense for everyday use) but I also know the other side of the communication channel and unfortunately altering existing tables and databases is not as easy as it seems because for this, programmer need to change lots of code of the app/program itself, not even talking of the problem of corrupting working data structures.

So you need to find a way in between. The problem is that users don't understand the programmers "language" and vice versa because they think on different levels. So it can happen they talk about the same thing but in their language and simply don't realize it and both get frustrated.

So I'm not saying that your idea is crap or that you don't know what you need, but simply that we need to find a way in between user and programming that programmers could easily apply to the existing structure. As I don't know their DB structure inside TM even my suggestions could be completely useless to them as it might not be able to add tables at all. ;)

regards,
Pascal

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Wed, 31 Jan 2018 23:00:24 GMT

Evzen Polenka Yes! This! This is exactly my worry and why I'm not enthusiastic about this solution.

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Wed, 31 Jan 2018 22:56:58 GMT

Pascal,

Thank you for your attempt. But unfortunately, your advice is not applicable.
Also, in response to "you don't need duplicate translations": I've said, repeatedly, that I, and my team, clearly do. If you're hung up on the word "duplicate" which I know the design team made trados on purpose to avoid duplicates, then think of it as "protection". I want certain data to be protected from being overwritten.

Your first suggestion, though I don't quite understand how you implement it, arrives at the impractical "cram multiple values in one field" solution, which I've explained before would result in huge fields. First of all, I'd have to turn "allow multiple field values option on" which would delete the contents of the field. I'm resigned to having to try this though using Daniel's import/export advice he gave me in the ideas group.

You wrote:
Alternatively, Trados could have an additional filter for TM result window could be set in project options:

Filter Fieldname: R11 manual

TM result window would then show results with Fieldname containing "R11 manual" as first result (most relevant one), which would mean less scrolling for the translator.

This wouldn't work because essential R11 TUs would've been overwritten by later data. So I can't use a filter to only see R11 data if some of that data doesn't exist (isn't associated with R11) anymore. In TM memory view, I've made changes to older files, filtered only what I needed, and noticed that once a batch edit was complete, the edited TUs always amount to LESS than what was originally added according to my records. So filtering doesn't work when data has been overwritten. Also, if I wanted to extract/export data from a TM using a field, I wouldn't actually get all the data I wanted because some data had been overwritten is no longer associated its original source.

RE: Translation memory management: duplicate TUs or overwritten TUs

Evzen Polenka — Wed, 31 Jan 2018 18:42:02 GMT

[quote user="Pascal Zotto"]

During translation it would look like this:

1 Source: パワーコンディショナー Target: power conditioner Filename: F11 manual 99% match (-1 penalty)
2 Source: パワーコンディショナー Target: PV inverter Filename: P11 manual; R11 manual 99% match (-1 penalty)

So the translators would see which TU to use for the manual they are translating whithout the TM result window being cluttered with too many strings.

[/quote]

Ugh, can you imagine the mess if there would be like 20+ filenames, 50+ characters each?!

RE: Translation memory management: duplicate TUs or overwritten TUs

Pascal Zotto — Wed, 31 Jan 2018 12:36:50 GMT

Hi Keenan,

I guess you are trying to put the cart before the horse.

You don't need multiple TU with same translation, which as Paul said is difficult to implement, but what about a workaround which might be way easier to implement?

Your example:

TU#1 Source: パワーコンディショナー Target: PV inverter Filename: P11 manual
TU#2 Source: パワーコンディショナー Target: PV inverter Filename: T11 manual
TU#3 Source: パワーコンディショナー Target: PV inverter Filename: R11 manual
TU#4 Source: パワーコンディショナー Target: PV inverter Filename: S11 manual
TU#5 Source: パワーコンディショナー Target: PV inverter Filename: Q11 manual
TU#6 Source: パワーコンディショナー Target: power conditioner Filename: F11 manual

So all six of those TUs are imported, in that order, meaning in that there are only two TUs in that TM:
TU#1 Source: パワーコンディショナー Target: PV inverter Filename: P11 manual
TU#6 Source: パワーコンディショナー Target: power conditioner Filename: F11 manual

Now I am translating something and I want it to be similar to the R11 manual, like, an update of the R11 manual.

You suggest you need several TU for the various Filename fields, so that translators know which one to use, but what if importing TU into your existing TM would have the following option (which checkbox to enable it):

If same source and target then update the following fields. (the fields would need to be selected with checkboxes)

in the background the process would use regex to check if field current string contains the string of the field from new TU and update it if necessary:

old value for field Filename:

P11 manual

value for field Filename of new TU:

R11 manual

new value for that TU in TM:

P11 manual; R11 manual

During translation it would look like this:

So the translators would see which TU to use for the manual they are translating whithout the TM result window being cluttered with too many strings.

Alternatively Trados could have an additional filter for TM result window could be set in project options:

Filter Fieldname: R11 manual

TM result window would then show results with Fieldname containing "R11 manual" as first result (most relevant one), which would mean less scrolling for the translator.

Regards,

Pascal

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Wed, 31 Jan 2018 01:43:12 GMT

Paul, Evzen Polenka

I added this option the ideas group. Let's hope it gets some traction...
community.sdl.com/.../allow-duplicates-or-create-write-protected-tus

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Wed, 31 Jan 2018 01:41:54 GMT

Yes, Paul. You got it.

RE: Translation memory management: duplicate TUs or overwritten TUs

Paul — Mon, 29 Jan 2018 09:48:55 GMT

Hi Evzen Polenka ,

You're not missing anything... there is no such option. I was suggesting Keenan might want to suggest this when/if he creates his idea. But re-reading what I have written I can see it wasn't very clear! Sorry about that.

RE: Translation memory management: duplicate TUs or overwritten TUs

Evzen Polenka — Mon, 29 Jan 2018 08:16:18 GMT

[quote user="Paul"]Maybe an option to suggest is to add new translation units if the field values differ[/quote]

How do I "add new translation unit" as an engineer when performing TM update after receiving translation packages from translators?
AFAIK, adding new TU is possible only during translation (i.e. on the translator's side), but that action is completely disconnected from what happens in the main project after importing the return package...
Or am I missing something? Is the "this segment is supposed to be added as new TU" information recorded somewhere in the SDLXLIFF (so that subsequent TM update can act accordingly)? I believe it's not.

RE: Translation memory management: duplicate TUs or overwritten TUs

Paul — Mon, 29 Jan 2018 07:39:24 GMT

Hi Keenan Cooper

Thanks for your email, and your patience, explaining to me again what your problem is. I'm going to summarise the situation so it's clear (for me at least):

you need to be able to differentiate between duplicate translations based on your field values
you have many field values in use, all of which are important, so using "allow multiple values" is not a solution for you because there would be too many in the TM results pane to make it possible to use them during translation
you could apply filters to the TMs based on a field value so you only get results from the field you want, but this is not a solution for you either
using multiple TMs is not a solution for you because the management of so many TMs is impractical
what you would like is the ability to update and create duplicates during import

Currently, as you know, it is very difficult by design to create a true duplicate in a Studio TM. Even if you go to TM Maintenance and manually create one by copy pasting over another you'll find Studio removes one automatically. So making the change to allow this would be an enhancement, and probably not a simple one either. It is also very likely to cause duplication where you don't want it because it would probably be an all or nothing approach and this would add unwanted maintenance to your TMs. Maybe an option to suggest is to add new translation units if the field values differ and perhaps that would give you the desired effect?

TM updates is quite a complicated area and all changes need to be very carefully thought through, however, I think the best approach is for you to log this request in the ideas site and the product management team can pick it up and decide whether it's something they wish to introduce based on their views and the amount of support you get.

I hope this reflects the situation. Sorry it took so long to get there too!

RE: Translation memory management: duplicate TUs or overwritten TUs

Paul — Sat, 27 Jan 2018 11:50:09 GMT

Hi Keenan Cooper

I spent a little time playing with this today after sending six packages out to different people so I could reproduce this as accurately as possible. I do have different context in the TU because I was getting confused with all the duplicates but you can hopefully still see what is happening. First, this is what I get if I update without any fields at all, as expected:

The F11 and others are treated as variables so even though my source for the manual names are different I only get one TU. Then as expected, and as you explained, I get two TUs for the two segments we are talking about. Five of them translated the same way, one different.

If I use fields, set up like this:

I used a picklist to control what I was getting, but you could use any type of field you like. The important part is that I'm allowing multiple values. When I update the TM with the returned packages and apply the appropriate update value into the fields then I get this in my TM:

So still four results but the fields values are all added. The effect this has in the files themselves would be this:

So I can see which one to choose based on the field name. Doesn't this approach work for you?

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Fri, 26 Jan 2018 01:38:04 GMT

Here's another example, similar to my earlier flier/pamphlet example.

TU#1 Source: パワーコンディショナー Target: PV inverter Filename: P11 manual
TU#2 Source: パワーコンディショナー Target: PV inverter Filename: T11 manual
TU#3 Source: パワーコンディショナー Target: PV inverter Filename: R11 manual
TU#4 Source: パワーコンディショナー Target: PV inverter Filename: S11 manual
TU#5 Source: パワーコンディショナー Target: PV inverter Filename: Q11 manual
TU#6 Source: パワーコンディショナー Target: power conditioner Filename: F11 manual

So all six of those TUs are imported, in that order, meaning in that there are only two TUs in that TM:
TU#1 Source: パワーコンディショナー Target: PV inverter Filename: P11 manual
TU#6 Source: パワーコンディショナー Target: power conditioner Filename: F11 manual

Now I am translating something and I want it to be similar to the R11 manual, like, an update of the R11 manual.

When I, or the several non-native translators in my company, come across パワーコンディショナー in the source, the translation memory results window shows results like this:
1 Source: パワーコンディショナー Target: power conditioner Filename: F11 manual 99% match (-1 penalty)
2 Source: パワーコンディショナー Target: PV inverter Filename: P11 manual 99% match (-1 penalty)

So which one does the translator choose? It's pretty much a toss-up. The vital TU#3 was overwritten so the translator is lost. If the first 5 TUs were not overwritten, the translator could see that "power conditioner" is the preferred and most used option, plus, most importantly, knowing the origin of the TU, would help translation when the context is just as important as the target.
So if the translator chooses "power conditioner" it would be wrong. Obviously, a TB could supplement this by defining context, which we do have, but I'm just giving this short TU as an example.
I just explained this to a coworker and he also strongly agreed that knowing the context is important. My other Japanese coworkers would freak if I told them their data was being overwritten.

RE: Translation memory management: duplicate TUs or overwritten TUs

Paul — Thu, 25 Jan 2018 11:23:44 GMT

[quote user="Keenan Cooper"]Also, from an earlier post, seeing multiple duplicates with different fields does make sense and does affect the translator and translation. Seeing 10 occurrences of the same source-target verses 1 instance but with a different target, helps the translator choose the right one. [/quote]

Hi Keenan Cooper

ok... can you elaborate on this for me please as I'm clearly missing something here. If the source and target results are all the same then why does it matter which one the translator picks?

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Thu, 25 Jan 2018 10:54:31 GMT

Paul,

I have robust fields (category, subcategory, file name, translator, native check, date added) and this only helps when the target is different. If the target is the same, the field values get overwritten by those of the most recent import. As I said in my first post, if I changed the TM settings to "allow multiple values" then, as indicated by the hazard icon, it would delete all the fields there to begin with which would be a disaster. (My UI settings are in Japanese but you get the idea)

"If the only reason you want a 100% duplicate where source and target are exactly the same is for counting TUs then you need to force the software to not make the default choice."

As I explained several times, that is not the reason I want this option. Specifically, see the very non-hypothetical scenario I explained in my last post. Also, from an earlier post, seeing multiple duplicates with different fields does make sense and does affect the translator and translation. Seeing 10 occurrences of the same source-target verses 1 instance but with a different target, helps the translator choose the right one.
Also, I don't know what you mean by "forcing the software to not make the default choice."

RE: Translation memory management: duplicate TUs or overwritten TUs

Paul — Thu, 25 Jan 2018 10:01:32 GMT

Hi Keenan Cooper

[quote user="Keenan Cooper"]Same source and target but the context is different. A different file.[/quote]

In this case shouldn't you be using fields and attributes where multiple values are not allowed? If the only reason you want a 100% duplicate where source and target are exactly the same is for counting TUs then you need to force the software to not make the default choice. In translation terms it makes no sense since the results will be the same with either seeing as the context doesn't change the translation at all.

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Thu, 25 Jan 2018 09:52:00 GMT

Paul,

"But in this scenario they are not duplicates, only the source is duplicated and if you use the option when importing to add as new translation if the target segments differ then you will get two and the new one will also retain the different fields."

In the scenario I presented, the source is the same and the target is the same. Nothing differs except the origin of the unit. That's what I need to keep. Constantly having the same TU overwrite itself over and over is no help when knowing the origin defines how a TU is used. This option would be really valuable for the type of translation we do.

"If they are both the same context then I think you should be questioning why you have duplicates in there in the first place."

Not quite. Same source and target but the context is different. A different file. As I explained before, having duplicates is beneficial because knowing the source of the TU is important. Here's another example. Say from an empty TM, I imported File #1 with 500 TUs. Then I imported File #2 with 500 TUs but 300 TUs were the exact same as File #1. So those 300 TUs from File #2 overwrote 300 of File #1's TUs. Right? So I only have 700 TUs when I actually need 1000. Then say I want to make a TM with just the TUs from File #1. If I filtered them out, I'd only have 200 TUs which is incomplete. This is not theoretical. This is something I will be doing but I risk having incomplete data because of how I didn't have control of how data was handled.

RE: Translation memory management: duplicate TUs or overwritten TUs

Paul — Thu, 18 Jan 2018 09:32:24 GMT

Hi Keenan Cooper

[quote user="Keenan Cooper"]Even better if there were field attached so you could see what manual this TU was used. So having this as an option when importing would be great.[/quote]

You can do this and then even import based on the fields if you like:

You might also find this app useful as it can ensure you never forget to set the right field for recording filenames:

https://multifarious.filkin.com/2016/01/14/recordsourcetu/

RE: Translation memory management: duplicate TUs or overwritten TUs

Paul — Thu, 18 Jan 2018 09:11:56 GMT

Hi Keenan Cooper ,

But in this scenario they are not duplicates, only the source is duplicated and if you use the option when importing to add as new translation if the target segments differ then you will get two and the new one will also retain the different fields. So when translating you would see a duplicate translation penalty but you'd also see the fields and could make your own mind up which one was correct.

Furthermore the TU carries a context with it, so if the TU was created in a file with a different context then Studio should be able to distinguish between the results and not give you the penalty. If they are both the same context then I think you should be questioning why you have duplicates in there in the first place.

Maybe I'm not understanding your problem properly, but I can import without overwriting in this scenario. The only problem I can't resolve is knowing which is the correct translation when the context is the same. But I think that's a challenge all users have in ensuring that their TMs contain accurate information, especially when working as a project manager with multiple translators who don't get to reference your main TM when they work.

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Thu, 18 Jan 2018 08:57:03 GMT

Paul,
Here's an example of why having duplicates improves reliability.

Say I have this data in one TM:

TU#1 Source: これはチラシです。Target: This is a pamphlet.
TU#2 Source: これはチラシです。Target: This is a flier.

If you were to come across "これはチラシです" while translating, the TM result window would show both of the above TUs with a -1 penalty. They'd both have a 99 score. The top one is only there because it was imported/added most recently, right? That's the only thing separating these two. So how do I know which one is correct? But what if say TU#2 had actually been imported/added a dozen times with the same source/target from a dozen different manuals with unique fields that when imported all just overwrote the previous TU? And say TU#1 was only used once. You'd like to know that a certain unit has been used many times with a TM right? Knowing this would improve reliability. Even better if there were field attached so you could see what manual this TU was used. So having this as an option when importing would be great. As said above, the option exists when translating to add a new TU, but this option doesn't exist when aligning/importing/batch updating main translation memory. If something had a different field value, I'd like a new TU to be created even if the source and target are the exact same. If I wanna know which translation was used for manual XX, I'd have no way of knowing because all critical field values are overwritten. The only way, as of now, to do this is to have a different TM for every manual, which gets outta hand. Or to set the TM to allow for duplicate values... but this would make huge, space-eating field values, plus would make me have to reimport all the TM data.
See what I'm getting at?

RE: Translation memory management: duplicate TUs or overwritten TUs

Paul — Thu, 18 Jan 2018 07:54:27 GMT

Hi Keenan Cooper ,

I am a little confused by this thread. Can you provide an example of where you would want duplicate TUs in your TM? If they are true duplicates (same source, same target, same context) then you would not want duplicates at all and reliability should not be in question. It would be helpful if you could provide a sample TM with your fields, and sample TMX with different fields and a small text that demonstrates why this is a problem.

At the moment it feels too theoretical for me. I played around with some TMs and importing myself and the more I play with it the more I don't see why you want duplicates if the fields are not being used to distinguish between TUs in the import. It's actually incredibly hard to create a true duplicate in a Studio TM.

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Mon, 15 Jan 2018 00:16:22 GMT

Also curious as to if the method of adding to a translation memory via "update main translation memory task". In the options for that batch task, there is an option for "Add new translation units" but that's only "if target segments differ"... what about when the target is the same? Would be so easy if that option existed to ALWAYS add a new unit, regardless of whether the target is the same or different. Interest to hear if Paul has any thoughts on this.

RE: Translation memory management: duplicate TUs or overwritten TUs

Keenan Cooper — Mon, 15 Jan 2018 00:10:40 GMT

Ms Matefi,
Thank you for your response. However, this only works if adding units via translation. A large amount of our TM data comes from imports from data recieved from translation companies or from alignments of past documents.

RE: Translation memory management: duplicate TUs or overwritten TUs

Evzen Polenka — Fri, 12 Jan 2018 10:08:51 GMT

What I don't get is how this process works if translator works completely "independently", i.e. works on translation package (i.e. works either with Project TM, or with local separate copy of Main TM).
Then the return package contains only the SDLXLIFFs... so is this "Add as New Translation" flag remembered somewhere in the SDLXLIFF, so that when engineer runs the Update Main TM batch task on his side (i.e. completely separately from translators!), the TUs are written to TM exactly as the translator originally intended?!

RE: Translation memory management: duplicate TUs or overwritten TUs

Ana-Maria Matefi — Fri, 12 Jan 2018 09:53:21 GMT

Hello Keenan Cooper ,

If you try to import a TM in another TM the TU's will be overwritten.

The solution for you will be to use the CTRL + SHIFT + U function while translating and confirming segments.

Also the below article might be useful:
gateway.sdl.com/communityknowledge

Wish you a great day.

Best Regards,
Ana