Sarah Dussen 

The message seems to be saying "you have TUs in your file that are not nl-BE".  The implications being that your project in Extract is nl-BE.  So if you change all your codes to something other than nl-BE you'll make things worse, not better.

Perhaps you can confirm the languages of your Extract project, and then confirm the languages in your original TMX?

Thank you for your reply Paul.

I'm not sure what you mean? The target language I used for the extract project was nl-BE, given that the tmx target language was also nl-BE.

I realise it's contradictory, but it was the solution that was suggested in the only Q&A I found, to the same problem I had, just with a different target language.

I would be grateful if you could give it a try. What would be a safe way to share my tmx to you privately?

Sarah Dussen 

Sure... pfilkin at sdl dot com

Thank you! I sent you the export yesterday by e-mail.

Sarah Dussen 

Thanks for the file... found some time to look at it this evening.  I get the same error you mentioned, but the application still extracts the terminology:

So, I split the file and kept splitting it until I found the problem.  It relates to the segment contents rather than the language codes so this was a misleading error message.  The segment (<seg>) for both the fr-BE and nl-BE languages contains placeholders (<ph>, <bpt>, and <ept>) but no actual translatable content in one single <tu>. The placeholders refer to certain tag formats or structures that were present in the original document.  If you're extracting terminology, the actual terms or sentences should be inside the <seg> tags and since this file you provided has a single TU with only placeholders and no content Extract seems to get a bit upset!

This is the offending translation unit variant:

      <tuv xml:lang="fr-BE">
        <seg><bpt i="1" type="110" x="1" /><ph x="2" type="114" /><ept i="1" /></seg>
      </tuv>
      <tuv xml:lang="nl-BE">
        <seg><bpt i="1" type="110" x="1" /><ph x="2" type="114" /><ept i="1" /></seg>
      </tuv>

If you search for that and delete the entire translation unit and then try again the error is not there.

Sarah Dussen 

Just to close this off... you can use this regex to quickly find segments like this... also segments with no content at all... and that should save some time trying to find the offending segments:

<seg>(?:\s*<(bpt|ept|ph)[^>]*\/?>\s*)*<\/seg>

Hi Paul

Thank you for your detective work!

You are right, the extraction works despite the bug on the small file (207 kb) with 161 TUs.

Hoewever it doesn't work on our actual database of +265.000 KB and +200.000 TUs I'm actually trying to extract from.

The regex you provided works, but only to find just one faulty TU at a time.

Do you know of a regex that could erase the faulty TU's in our big extraction file from <tu to </tu>?

Sarah Dussen 

Sarah Dussen said:

Do you know of a regex that could erase the faulty TU's in our big extraction file from <tu to </tu>?

Do I know of one... no ;-) Can I create one... yes.  But it may not work for all files as it could be dependent on the specific format of your TMX files.  So based on the sample you sent me this seems to work.

Search for this:

<tu[^>]*>(?:(?!<\/tu>).)*(<seg>(?:\s*<(?:bpt|ept|ph)[^>]*\/?>\s*)*<\/seg>|<seg>\s*<\/seg>)(?:(?!<\/tu>).)*<\/tu>

Replace with nothing!

Hm.. We're not quite there yet ...

I tried the string (in Notepad++ I have to add, search mode Regular expression), and 0 results were found.

That shouldn't be possible as it includes the tu of the sample I sent you.

Do you have any more ideas?

Sarah Dussen 

It's often troublesome to use something without fully understanding it!  Make sure you check the "dot matches newline" checkbox... seems to work just fine:

That may be, but I'm determined to find a solution, so willing to learn trying to understand!

I'm getting this error message now:

That may be, but I'm determined to find a solution, so willing to learn trying to understand!

I'm getting this error message now:

Sarah Dussen 

And can you paste the regular expression you used in here so we can see it exactly.  Clearly there is nothing wrong with it for Notepad++ or I would have had the same message.  This is what I used:

<tu[^>]*>(?:(?!<\/tu>).)*(<seg>(?:\s*<(?:bpt|ept|ph)[^>]*\/?>\s*)*<\/seg>|<seg>\s*<\/seg>)(?:(?!<\/tu>).)*<\/tu>

Sarah Dussen 

I don't really use Notepad++ but I deliberately broke my expression to see what that little note at the end of the message does and it seems to tell you what's wrong with it if you hover over it:

Maybe share that... or use it to fix your own problem?

Sarah Dussen 

Maybe your Notepad++ version is too old. Latest version is 8.5.6 and fixes are implemented every few days.

Try upgrading from https://notepad-plus-plus.org/downloads/.

Thank you for that suggestion Jesús.

I installed version 8.5.6 only yesterday.

This is the message I'm getting:

Sarah Dussen 

ok - so you must be running this on your larger file?

I think the error is probably due to the complexity of the regular expression, which is taking too long to evaluate.  Regular expressions that have lots of nested repetitions or non-greedy matches can be highly inefficient, especially on large texts.  You may fare better with another editor... I use EditPadPro which handles far larger files than notepad++ can work with, but even with this I don't know how well it would work with your specific file.  You'd need to test and see.

I also don't think I know how to simplify this expression with regex alone as you'd have to start getting too specific and this would remove the benefit of the regex in the first place, so you may be better off using another method to clean up such large files.  Maybe Powershell or Python for example which can make use of some built in libraries that make it easier to manipulate XML.