Youngrok Kim we made major changes on the editor in this new SR. One of the big changes is in text analysis and word detection. One of the criteria to detect a word is the ability of the glyph to wrap around, the ability for the word to be broken at the end of the line. In Latin based languages we use a hyphen, so i's very specific and user driven, but I see in Korean I get that all glyphs can wrap around (or at least all I tried).

My knowledge of Korean is very limited and what I found online is not very concluding.

Could you confirm that words can be wrapped around so that I remove this criteria from the code?

Bogdan Costisor Youngrok Kim 

If it's interesting... here's a little AI interpretation of the answer to your question.

Yes, in Korean, words can indeed be wrapped around to the next line without the need for hyphens. Korean is an agglutinative language, which means that words are often formed by combining smaller morphemes (units of meaning), but unlike in some other agglutinative languages, there's no hard and fast rule in Korean about not splitting these words across lines. You can break a line after any character, except some punctuation marks.

This makes sense in the context of the Korean writing system, Hangul, which is composed of individual syllable blocks. Each block is like an independent character and contains one to three smaller elements, each of which represents a specific sound.

Therefore, removing the word-wrapping criteria for Korean in your code should be appropriate. However, please note that depending on the context, splitting certain words across lines could potentially introduce confusion or difficulty in reading, especially for compound words or words composed of multiple morphemes. Therefore, it's generally a good practice to try to keep words intact whenever possible, even if it's not strictly required by the language rules.

Good to validate this from a Korean native speaker though :-)

Paul amazing answer from an AI:) To be fair I don't think we need this criteria for any language. For Latin languages I also get the flag that it's preceded by a soft hyphen, so in actuality I think we are safe to completely remove it . I get a lot of pieces of information after a text analysis, in this case I thought it would enhance word detection, but it should only be kept for line wrapping.

The new editor is language agnostic so far and I hope to keep it that way. 

The features related to line breaks in Korean are: 1) line breaks are possible anywhere in a sentence, except for most of punctuation marks, parentheses, etc.; 2) when a line break occurs, the original space should be maintained, even if a space occurs at the end of a line and the space at the beginning of the next line is not visible.

Korean basically forms basic character as a combination of initial consonant + neutral vowel (+ final consonant). For example, “초” is “ㅊ” + “ㅗ or 오”, and “성” is “ㅅ” + “ㅓ or 어” + “ㅇ”.

Nowadays, various forms are used in Korea, such as writing only the initial consonant. For example, "하이요" corresponding to Hello can also be expressed as "ㅎㅇㅇ" or "ㅎ이요" or "ㅎㅇ요".

Further, Korean is an agglutinative language. It is a type of language that indicates grammatical relationships or functions by adding suffixes (called "postpositions" or Korean pospositions), which are grammatical forms, to substantive forms, which are semantic parts. Korean, Turkish, and Japanese are typical examples, and in these added languages, grammatical elements such as functional markers and sentence ending forms are added to the actual form to clearly indicate grammatical relationships or structural meanings (See. Suh Cheong-Soo, "A Korean Grammar," Jipmoondang, March, 2013).

In addition, there are a lot of Korean pospositions in Hangeul, and the number of Korean pospositions collected in the book I currently have is about 2,200. So, extracting words from Hangul sentences seems to require a lot of effort. For reference, in MS Word's spelling and grammar checking tool, phrase must be registered whenever Korean posposition is changed for the same word.

Lastly, please consider that this is an content written in Korean and roughly edited after running Google Translate.

Youngrok Kim thank you for your reply. We managed to develop and test a solution that fixes not only Korean, but also Thai and probably other languages that work similarly (as you already mentioned Turkish and Japanese might also have this issue).

According to Wikimedia, moving the cursor (caret) using arrows in a text editor or word processor is called caret navigation.

I don't know what the caret (or cursor) navigation in the text editor window has to do with word tokenization. However, what is clear is that editing text in the Trados editing window has become very inconvenient for users, owing to the cursor navigation error.

I hope the word tokenization works well in Korean. However, please fix the cursor navigation error as soon as possible.

When you press ctrl+arrow key you navigate from one word to the next/previous. We need to determine where a word start and ends so that we move the caret accordingly. In our scenario, we considered each Korean character a word, this being the root of the problem.
You would have seen the same problem if you decided to double click and select a word.

I don't know how wikimedia decided to define a caret navigation, but it is more complex than simply moving somewhere a cursor. A vast text analysis is required to determine locale, language, is it bidirectional, does it have any punctuation mark next to it, does it have any ligature, is the ligature treated separately or combined and the list can continue.

It might sound trivial as simply moving from one character to another, but in reality the decision where to move is extremely complex.

When you press ctrl+arrow key you navigate from one word to the next/previous. We need to determine where a word start and ends so that we move the caret accordingly. In our scenario, we considered each Korean character a word, this being the root of the problem.
You would have seen the same problem if you decided to double click and select a word.

I don't know how wikimedia decided to define a caret navigation, but it is more complex than simply moving somewhere a cursor. A vast text analysis is required to determine locale, language, is it bidirectional, does it have any punctuation mark next to it, does it have any ligature, is the ligature treated separately or combined and the list can continue.

It might sound trivial as simply moving from one character to another, but in reality the decision where to move is extremely complex.

Korean is an agglutinative language with spacing, and Korean particles (called Korean postpositions) are placed and changed at the end of morphemes, as in inflecting languages such as English. So, in a text editor or word processor that supports Korean, such as MS Word, the carat navigation in Korean sentence is the same as in English sentence. This means that in a sentence with white spaces, letters, punctuation marks, etc., moving the caret with Ctrl+left or right arrow key is substantially the same in Korean and English.

MS Word's use of the same carat navigation in Korean and English is considered to be for the case where Korean sentences or phrases and English sentences or phrases are mixed in a document.

I believe that Trados continued to use a method similar to the above caret nevigation until version 2022. And as tested, even in the case of Trados 2022 SR1, in the input line of the Find and Replace window in the Trados studio Editor, each time Ctrl + left (or right) arrow key is pressed, the caret is moved from a position between a space and a letter to a new position between another space and another letter in the korean sentence.