The Unicode Consortium Discussion Forum

I'm wondering if someone can help me.

I am trying to convert some data coming from an IBM iSeries system encoded in EBCDIC (Arabic CCSID 420) to UTF-16. The original data consists of a date, some Arabic text and a currency amount, all stored (as is usual for Arabic EBCDIC data on iSeries ) in visual sequence, e.g.:

01/10/12 TXET CIBARA 3,902.07

I am successfully converting this using ICU to UTF-16 and then applying bidi processing to produce UTF-16 data that has the Arabic text stored in logical sequence:

01/10/12 ARABIC TEXT 3,902.07

HOWEVER, when this is displayed by NotePad, what the user sees is:

01/10/12 3,902.07 TXET CIBARA

In other words, the Arabic is display correctly (RTL) but the currency amount has moved to the left of it. This is not acceptable to my users, who require the order of the items (date, text, amount) on the page to remain unchanged.

I am guessing this is happening because of this logic, described by the Unicode bidi spec http://unicode.org/reports/tr9/#BD1:



My question is: can I override this logic somehow, e.g. through the use of format codes, to ensure that the data items (date, Arabic text, currency amount) appear in the correct sequence, but the Arabic is still displayed correctly? I have so far failed to find any set of format codes that makes this happen.

Thanks in advance for any assistance you can offer.

What you are trying to do is to transform a visual string into a logical LTR string. However, there are visual strings which cannot have an equivalent logical string without adding a directional mark, in this case the LRM (U+200E).
The most frequent such case is when a number appears to the right of RTL text and to the left of LTR text (or to the left of the end-of-string in your case).
To fix it, you have to add a LRM before the number after the conversion to UTF-16 and before calling ICU to transform the string to logical.
You can do this automatically if when calling ICU to transform the string from visual to logical you set the reordering mode to UBIDI_REORDER_INVERSE_LIKE_DIRECT, and set the reordering option to UBIDI_OPTION_INSERT_MARKS, if you use ICU4C, or the equivalent REORDER_INVERSE_LIKE_DIRECTand OPTION_INSERT_MARKS if you use ICU4J.

Thank you for your very helpful response. We will explore some of the options you mention and will let you know if we are able to resolve our issue.