Also curious as to if the method of adding to a translation memory via "update main translation memory task". In the options for that batch task, there is an option for "Add new translation units" but that's only "if target segments differ"... what about when the target is the same? Would be so easy if that option existed to ALWAYS add a new unit, regardless of whether the target is the same or different. Interest to hear if Paul has any thoughts on this.

Paul,
Here's an example of why having duplicates improves reliability. 

Say I have this data in one TM:

TU#1 Source: これはチラシです。Target: This is a pamphlet.
TU#2 Source: これはチラシです。Target: This is a flier.

If you were to come across "これはチラシです" while translating, the TM result window would show both of the above TUs with a -1 penalty. They'd both have a 99 score. The top one is only there because it was imported/added most recently, right? That's the only thing separating these two. So how do I know which one is correct? But what if say TU#2 had actually been imported/added a dozen times with the same source/target from a dozen different manuals with unique fields that when imported all just overwrote the previous TU? And say TU#1 was only used once. You'd like to know that a certain unit has been used many times with a TM right? Knowing this would improve reliability. Even better if there were field attached so you could see what manual this TU was used. So having this as an option when importing would be great. As said above, the option exists when translating to add a new TU, but this option doesn't exist when aligning/importing/batch updating main translation memory. If something had a different field value, I'd like a new TU to be created even if the source and target are the exact same. If I wanna know which translation was used for manual XX, I'd have no way of knowing because all critical field values are overwritten. The only way, as of now, to do this is to have a different TM for every manual, which gets outta hand. Or to set the TM to allow for duplicate values... but this would make huge, space-eating field values, plus would make me have to reimport all the TM data. 
See what I'm getting at?

Hi Keenan Cooper 

Unknown said:

Even better if there were field attached so you could see what manual this TU was used. So having this as an option when importing would be great.

You can do this and then even import based on the fields if you like:

You might also find this app useful as it can ensure you never forget to set the right field for recording filenames:

https://multifarious.filkin.com/2016/01/14/recordsourcetu/

Paul,

"But in this scenario they are not duplicates, only the source is duplicated and if you use the option when importing to add as new translation if the target segments differ then you will get two and the new one will also retain the different fields."

In the scenario I presented, the source is the same and the target is the same. Nothing differs except the origin of the unit. That's what I need to keep. Constantly having the same TU overwrite itself over and over is no help when knowing the origin defines how a TU is used. This option would be really valuable for the type of translation we do. 

"If they are both the same context then I think you should be questioning why you have duplicates in there in the first place."

Not quite. Same source and target but the context is different. A different file. As I explained before, having duplicates is beneficial because knowing the source of the TU is important. Here's another example. Say from an empty TM, I imported File #1 with 500 TUs. Then I imported File #2 with 500 TUs but 300 TUs were the exact same as File #1. So those 300 TUs from File #2 overwrote 300 of File #1's TUs. Right? So I only have 700 TUs when I actually need 1000. Then say I want to make a TM with just the TUs from File #1. If I filtered them out, I'd only have 200 TUs which is incomplete. This is not theoretical. This is something I will be doing but I risk having incomplete data because of how I didn't have control of how data was handled. 

Hi Keenan Cooper 

Unknown said:

Same source and target but the context is different. A different file.

In this case shouldn't you be using fields and attributes where multiple values are not allowed?  If the only reason you want a 100% duplicate where source and target are exactly the same is for counting TUs then you need to force the software to not make the default choice.  In translation terms it makes no sense since the results will be the same with either seeing as the context doesn't change the translation at all.

Paul,

I have robust fields (category, subcategory, file name, translator, native check, date added) and this only helps when the target is different. If the target is the same, the field values get overwritten by those of the most recent import. As I said in my first post, if I changed the TM settings to "allow multiple values" then, as indicated by the hazard icon, it would delete all the fields there to begin with which would be a disaster. (My UI settings are in Japanese but you get the idea)

"If the only reason you want a 100% duplicate where source and target are exactly the same is for counting TUs then you need to force the software to not make the default choice."

As I explained several times, that is not the reason I want this option. Specifically, see the very non-hypothetical scenario I explained in my last post. Also, from an earlier post, seeing multiple duplicates with different fields does make sense and does affect the translator and translation. Seeing 10 occurrences of the same source-target verses 1 instance but with a different target, helps the translator choose the right one.
Also, I don't know what you mean by "forcing the software to not make the default choice."

Unknown said:

Also, from an earlier post, seeing multiple duplicates with different fields does make sense and does affect the translator and translation. Seeing 10 occurrences of the same source-target verses 1 instance but with a different target, helps the translator choose the right one.

Hi Keenan Cooper 

ok... can you elaborate on this for me please as I'm clearly missing something here.  If the source and target results are all the same then why does it matter which one the translator picks?

Unknown said:

Also, from an earlier post, seeing multiple duplicates with different fields does make sense and does affect the translator and translation. Seeing 10 occurrences of the same source-target verses 1 instance but with a different target, helps the translator choose the right one.

Hi Keenan Cooper 

ok... can you elaborate on this for me please as I'm clearly missing something here.  If the source and target results are all the same then why does it matter which one the translator picks?

Here's another example, similar to my earlier flier/pamphlet example.

TU#1 Source: パワーコンディショナー Target: PV inverter    Filename: P11 manual
TU#2 Source: パワーコンディショナー Target: PV inverter    Filename: T11 manual
TU#3 Source: パワーコンディショナー Target: PV inverter    Filename: R11 manual
TU#4 Source: パワーコンディショナー Target: PV inverter    Filename: S11 manual
TU#5 Source: パワーコンディショナー Target: PV inverter    Filename: Q11 manual
TU#6 Source: パワーコンディショナー Target: power conditioner    Filename: F11 manual

So all six of those TUs are imported, in that order, meaning in that there are only two TUs in that TM:
TU#1 Source: パワーコンディショナー Target: PV inverter    Filename: P11 manual
TU#6 Source: パワーコンディショナー Target: power conditioner    Filename: F11 manual

Now I am translating something and I want it to be similar to the R11 manual, like, an update of the R11 manual.

When I, or the several non-native translators in my company, come across   パワーコンディショナー in the source, the translation memory results window shows results like this:
1 Source: パワーコンディショナー Target: power conditioner    Filename: F11 manual 99% match (-1 penalty)
2 Source: パワーコンディショナー Target: PV inverter    Filename: P11 manual 99% match (-1 penalty)

So which one does the translator choose? It's pretty much a toss-up. The vital TU#3 was overwritten so the translator is lost. If the first 5 TUs were not overwritten, the translator could see that "power conditioner" is the preferred and most used option, plus, most importantly, knowing the origin of the TU, would help translation when the context is just as important as the target. 
So if the translator chooses "power conditioner" it would be wrong. Obviously, a TB could supplement this by defining context, which we do have, but I'm just giving this short TU as an example. 
I just explained this to a coworker and he also strongly agreed that knowing the context is important. My other Japanese coworkers would freak if I told them their data was being overwritten.

Hi Keenan Cooper 

I spent a little time playing with this today after sending six packages out to different people so I could reproduce this as accurately as possible.  I do have different context in the TU because I was getting confused with all the duplicates but you can hopefully still see what is happening.  First, this is what I get if I update without any fields at all, as expected:

The F11 and others are treated as variables so even though my source for the manual names are different I only get one TU.  Then as expected, and as you explained, I get two TUs for the two segments we are talking about.  Five of them translated the same way, one different.

If I use fields, set up like this:

I used a picklist to control what I was getting, but you could use any type of field you like.  The important part is that I'm allowing multiple values.  When I update the TM with the returned packages and apply the appropriate update value into the fields then I get this in my TM:

So still four results but the fields values are all added.  The effect this has in the files themselves would be this:

So I can see which one to choose based on the field name.  Doesn't this approach work for you?

Hi Keenan Cooper 

Thanks for your email, and your patience, explaining to me again what your problem is.  I'm going to summarise the situation so it's clear (for me at least):

• you need to be able to differentiate between duplicate translations based on your field values
• you have many field values in use, all of which are important, so using "allow multiple values" is not a solution for you because there would be too many in the TM results pane to make it possible to use them during translation
• you could apply filters to the TMs based on a field value so you only get results from the field you want, but this is not a solution for you either
• using multiple TMs is not a solution for you because the management of so many TMs is impractical
• what you would like is the ability to update and create duplicates during import

Currently, as you know, it is very difficult by design to create a true duplicate in a Studio TM.  Even if you go to TM Maintenance and manually create one by copy pasting over another you'll find Studio removes one automatically.  So making the change to allow this would be an enhancement, and probably not a simple one either.  It is also very likely to cause duplication where you don't want it because it would probably be an all or nothing approach and this would add unwanted maintenance to your TMs. Maybe an option to suggest is to add new translation units if the field values differ and perhaps that would give you the desired effect?

TM updates is quite a complicated area and all changes need to be very carefully thought through, however, I think the best approach is for you to log this request in the ideas site and the product management team can pick it up and decide whether it's something they wish to introduce based on their views and the amount of support you get.

I hope this reflects the situation.  Sorry it took so long to get there too!

Unknown said:

Maybe an option to suggest is to add new translation units if the field values differ

How do I "add new translation unit" as an engineer when performing TM update after receiving translation packages from translators?
AFAIK, adding new TU is possible only during translation (i.e. on the translator's side), but that action is completely disconnected from what happens in the main project after importing the return package...
Or am I missing something? Is the "this segment is supposed to be added as new TU" information recorded somewhere in the SDLXLIFF (so that subsequent TM update can act accordingly)? I believe it's not.

Hi Evzen Polenka ,

You're not missing anything... there is no such option. I was suggesting Keenan might want to suggest this when/if he creates his idea.  But re-reading what I have written I can see it wasn't very clear!  Sorry about that.