The thing is that in order to be able "to handle tags" (i.e. "to handle some generic tokens") you need to be able "to identify the tags/tokens" in the first place... i.e. to be able to identify somehow that "this is a tag/token".
And this is not - or should not - be possible currently. Simply because there isn't any built-in way to identify "tag/token" in the regexes... they work just a standard regexes, without any such additional functionality.
As I said, the fact that the pointy brackets constructs currently work for you is apparently just coincidence and not intentional... and actually a BUG, since that actually prevents from filtering segments containing only actual text enclosed in pointy brackets, without polluting the results by segments containing tags, but not text in pointy brackets!.

I do understand that the current functionality is very useful for you and I do understand why. Am just saying "enjoy it while you still can"... but (hopefully!) do NOT expect this apparently incorrect behavior to be carried over to regex QA checker.

Hi Evzen,

I agree with you that using pointy brackets for identifying tags is not a good idea.

For example, if I am searching for "foo bar", I will only find it if there are no unexpected tags in the text. A segment with "foo UNEXPECTED TAG bar" will not be found by the so-called "Advanced" display filter.

But there was an outpouring of relief by Studio users in another thread. They were so happy that they had a way to identify tags. As a result, I limited myself to suggesting that a "Tag" button be added so that you could specify whether or not you wanted to identify tags.

If you did not want to identify tags, then tags would be deleted before the text was sent for regex processing. If you wanted to identify tags, then tags would be converted to "<tag text>" before sending for regex processing. So "foo TAG bar" would be converted to "foo bar" or "foo <tag> bar" depending on the "Tag" button.

Adding a "Tag" button would also inform Studio users that the "Advanced" display filter was doing something different. The way things are now, unless you read one of these threads there is no way you could possibly know that the "Advanced" display filter is unable to reliably find "foo bar".

The current flawed solution is popular, so I doubt it will disappear. I think it will very likely be expanded into other areas where regex is used.

SDL is probably also happy with this solution because they only have to modify the text sent for regex processing, not the actual regex code itself. For example, using "\<" and "\>" to identify the start and end of tags would require modifying the regex code, which could introduce bugs into regex processing -- if it was not done carefully.

Another possibility is outlined below. I have not thought this through fully, but imagine that we produce two versions of the input string, one without tags and one with tags, using "<" and ">" to delimit the tags.

Then start matching the string without tags against the regex expression until it fails. If it fails because the regex expression is expecting a tag, then switch to the version of the input string that includes tags and see whether this string satisfies the regex expression.

If the string with tags does match, then continue checking the match until either the end of the string or the end of the tag. At the end of the tag, switch back to the string without tags and continue matching until the match fails. If it fails because the regex expression is expecting a tag, then ... and so on.

From my point of view, this has the advantage that "unexpected" tags are simply ignored. After all, if you want to match a tag, then you presumably know where it is going to be and can include it properly in your regex search expression. You cannot predict where any unexpected tags might occur in your text, so there is no way you can include them in a regex expression.

So if I am looking for "foo bar TAG1", the "Advanced" display filter will currently not find "foo UNEXPECTED TAG bar TAG1".

However, the solution I outlined above would find a match.

For example, if we create two versions of the input string, namely "foo bar" and "foo <unexpected tag> bar <tag1>" and then start matching "foo bar" against the regex expression "foo bar <tag1>", then the match will not fail until "<tag1>" is reached. This means that the "<unexpected tag>" section of the second input string is just ignored, since we are processing the first string at that point. When the match fails, we switch to the second string and start matching again. Since the regex expression is looking for "<tag1>" at this point, and that is precisely what is in the second string, the regex expression is considered a match.

Given the unpredictable location of many tags in the files I work with, an approach like this is the only proper way I can imagine for handling them.

Of course, someone else might say that they only want to find segments that are exact matches, i.e. they don't want to skip over anything. So they only want the match to succeed on "foo bar TAG1" and fail on "foo UNEXPECTED TAG bar TAG1".

Finding a solution that handles every conceivable situation is the difficulty here. Do we need another button: "Ignore the unexpected" ??

Hopefully SDL will consider questions like these before they go any further with regex tag matching.

Best regards,
Bruce Campbell
ASAP Language Services

It's no surprise it is a useful feature. As things are now, we can only do a few things with tags, namely, check their existance, order and spacing. As important as they are, we can't do nothing more with them. We cannot check whether they are incorrectly placed with a regular expression, for example, so I, for one, would go for a way to be able to include them in all possible Trados features: Advanced Display Filter, Searches and Regex QA. Advanced Display filter does not really work well, as I have NOT been able to find a way, for example, to detect target segments ending in a period and a tag. I don't remember the exact necessity I had right now so I can't show you the exact difficulty I had, but it's true that Advanced Display Filter works well for basic searches, but if you want to expand the possibilities, it tends to fail.

I, for one, think that your tag button solution would working well, as long as "FOO UNEXPECTED TAG1 BAR" can be matched by just a default "FOO BAR" search. I checked, and it's as you say, if there is an unexpected tag, Advanced Display Filter will NOT find the string "FOO BAR".

Imagine the possibilities if we could integrate tags to our Regular Expressions in a successful way? ADF is just flawed, because it adds a very interesting functionality, but wrongly and incompletely.

I now remember the problem I had with ADF. I wanted to find source segments that ended in a tag and BEFORE that, NOT a period. In other words, segments that ch as this:

For that, I used this 

And it does not work, as it still does match segments such as this one as well.

so its behaviour is quite erratic and including tags in ADF is not really well implemented. But remember, I want to be able to use Tag identification BOTH in Advanced Display Filter AND in QA Regular Expression checks!!!

Hi  , Evzen Polenka ,

I don't intend to get drawn into this conversation too much (need some time to digest it properly as you guys write so much ;-)), but I think you are probably correct wrt the Advanced Display Filter. It was never intended to find tags in the way you are suggesting. But we do have the Community Advanced Display Filter where we are adding things all the time and we have actually added an option to filter on tags. It doesn't search inside the tags (although if this is what people want we, or any other developer, could add it as it is opensource) but it allows you to filter on segments with tags. I have not published it yet because Dan Lucas raised an issue with the colour filtering which we are trying to resolve (proving difficult as the API seems to be inconsistent in how it handles segment information in this regard and we want to avoid using the sdlxliff if we can), but I may publish the new things we already added anyway in the meantime. Perhaps at the end of the day.

Regards

Paul

Unknown said:

an issue with the colour filtering which we are trying to resolve (proving difficult as the API seems to be inconsistent in how it handles segment information in this regard and we want to avoid using the sdlxliff if we can)

So I hope that internal high-priority ticket for immediate fix of that inconsistency has been already logged...