Hi, I've gone a little deeper and have seen that tokenizer is more like the word count and it is certainly to tokenize the segment to check the similarity with existing translation units.

But the question remains the same: how can we set these flags. The function does not seem deprecated and on our server, we have 3 different sets: noFlags, DefaultTags and AllFlags. How is this possible?

Can you help? Romulus Crisan (former member)?

Kind regards

Sébastien Desautel

Can you please let me know which API you are using?

Hi Romulus,
It's in the translation memory API

here the link to the flag enumeration.
http://producthelp.sdl.com/SDK/TranslationMemoryApi/4.0/html/d8cc81a4-8731-9013-55b2-bf83f1f8b000.htm

Thank you

Sébastien

Hi Sébastien, 

       The TokinezerFlags are impacting the word count flags. The correct value that you need to set on the language resource template of the TM is DefaultFlags. Recognizer, tokenizer and word count flags need to be set on the template of the TM. In your case you can either create a template and use that across the TM's or create the TM save it in the server that will create a private TM template associated with the TM and set the flags. You can set the flags on the TM : ServerBasedTranslationMemory has all the flags that will update the private language resource template. 

       I hope that shared some light on your questions. 

Best regards,

Robert

Hi Robert,

thank you for your answer. Yes that's mainly what I wanted to know. I will set the tokenizer on "defaultFlags" for every TM container.

I guess the noFlags and allFlags we found for some containers are due to old TMs in older version of Studio where the property did not exist yet. Or is there any possiblity to set other Tokenizer Flags without using the API?

Kind regards

Sébastien

Hi Sébastien,

      As far as I know not really the flags are more the responsibility of the underlying components but this defaults changed over the time so it might be that. 

Best regards,

Robert

Hello Robert Cornea and Romulus Crisan (former member)

I want to share some troubleshooting we made recently by correcting the TokenizerFlags. This might be important for you to know and I 've just read in the forum that some people maybe need this information.

We faced 2 problems recently:

1. for a particular client, we received WorldServer packages that we analyze again with SDL Studio, one first time with the local TMs from the packages and one more time with our own Server TM. We had 3 different analyses! After some investigations we noticed, that the TokenizerFlags was set to "noFlags" and we repeated the analyse with another TM with "defaultFlags" and got the same result as the analyze with local TMs.

2. a couple of days later, before we corrected the TokenizerFlags of our own server TM, we got an error in the editor preventing the user to work on the file. The reason was that the number of confirmed words was higher as the number of words in the file. This was the result again, of the 2 different count methods.

Since we've corrected the TokenizerFlags in the client TM, we haven't got any problem anymore.

What's still disturbing, is that WorldServer still gives a different analyze but we haven't the possibility to investigate on this.

Kind regards

Sébastien

Hi Sébastien,

       One possible reason for the difference between WordSever and Studio might be that Studio has the latest fixes and improvements. This might cause differences between Studio and other on premise applications like WorldServer or GroupShare. Is your WorldServer instance up to date ?

Best regards,

Robert

Hi Robert,

yes the program versions may be the problem. The WS packages we receive are from WS 11.4.1 (info found in the package, it's not our own server). Can the reason be that we still use Studio 2017? We haven't made the step to 2019 and the migration to 2021 will only come in Q1 2021.

Thank you

Sébastien

Hi Sébastien,

        The releases are not that far apart if you are using Studio 2017 CU 18, but I can say for sure that the components responsible for the counting are the same. If you plan to migrate to 2021 I would to a test to see if there are any notable differences. 

Best regards,

Robert