Accumulated Feedback on PRI #231

This page is a compilation of formal public feedback received so far. See Feedback for further information on this issue, how to discuss it, and how to provide feedback.

Date/Time: Sun Jul 8 11:37:00 CDT 2012
Name: C. E. Whitehead
Report Type: Public Review Issue
Opt Subject: PRI 231 Bidi Parentheses Algorithm

Sorry I've not had time to look at this issue in depth (sick pet
comes 1rst; so these days does my online blog; sorry); my experience
at facebook suggests it's a good idea to have parentheses paired;
however a note of caution: doing so could affect "doodles" that make
use of punctuation characters such as ): [unhappy face] or (: [happy
face]. I do plan to look at the proposal further; there might be a
"work around" for the doodling issue.  --C. E. Whitehead

Date/Time: Wed Jul 11 12:04:05 CDT 2012
Name: C. E. Whitehead
Report Type: Public Review Issue
Opt Subject: PRI #231 (Bidi Parentheses Algorithm)

1rst, my apologies for mentioning (: :) :( ): these are not paired parens normally 
and would be ignored by the algorithm!!! That's great!
On the same note, I am not completely sure about { };
please see discussion of single curly braces in legal documents:
(Obviously though in such cases this rule will just not be implemented so I guess 
such uses are no problem again!).
Thus go ahead with all four kinds of braces:
 (), [], <>, and {}.
(I am not sure about others).

** IMO, the algorithm should be part of Unicode's core, 
as Unicode core's previous way of handling braces should be improved/corrected in the core,
even though (IMO again) the current rules HL4 and HL5 are also fine,
and do enable applications to fix/tweak the basic bidi algorithm.
What I mean is that, since if an app ignores HL4 and HL5 and simply applies the Unicode 
bidi algorithm the result could be mismatched brackets (in some cases);further, the 
rules in the core should be fixed in the core (IMO again) so long as they are only 
fixed for marks that are universal, and not language specific.
** Also I agree it's best to locate the matching brackets before applying the rest 
of the bidi algorithm.

Third, 3.1. Some comments on the algorithm itself:
"If an open parenthesis is found, push it onto a stack and continue the scan. If 
a close parenthesis is 85 found, check if the stack is not empty and the close 
parenthesis is the other member of the mirrored pair for the character on the 
top of the stack. If so, pop the stack and continue the scan; else return failure. 
If the end of the paragraph is reached, return success if the stack is empty; 
else return failure. Success implies that all open and close parentheses, if 
any, in a paragraph are matched correctly. Failure implies that there are one 
or more mismatched paired punctuation marks in a run and therefore the 90 handling 
under the parenthesis algorithm will not be attempted."
** The above is fine, IMO
"The rationale for following the embedding level in the normal case is that 
the text segment enclosed by 120 the paired punctuation marks will conform 
to the progression of other text segments in the writing direction. In the 
exception cases, the rationale to follow the opposite direction is based on 
context being established between the enclosed and adjacent segments with 
the same direction."
** Agreed, yes, embedding level should be followed in normal case, albeit 
for both brackets.
"Other neutral types adjacent to paired punctuation marks are resolved subsequent 
to resolving the paired punctuation marks themselves, and will therefore be 
influenced by that resolution."
** Agreed again, yes, so far so good.
"The directionality of the enclosed content is opposite the embedding direction, 
and at least one 115 neighbor has a bidi level opposite to the embedding 
direction O(O)E, E(O)O, or O(O)O."
"*N0. Paired punctuation marks take the embedding direction if the enclosed 
text contains a strong type of the same direction. Else, if the enclosed text 
contains a strong type of the opposite direction and at least one external 
neighbor also has that direction the paired punctuation marks take the direction 
opposite the embedding direction."
** I disagree with the above statements. 

R(R)L à R -- that is, with embedding ltr -- o.k.
L(R)R à R -- that is, with embedding ltr -- No; these brackets should 
take the ltr directionality, that is should take the directionality of 
the embedding level if same directionality immediately precedes opening 
paren (IMO again but see examples below).

** Same problem in an rtl embedding environment:
L(L)R à L with embedding rtl -- O.k. the text that precedes the parens 
is ltr, as is the text in parens, so fine, let the directionality be 
different from that of embedding directionality.
R(L)L à L with embedding rtl -- No; same problem as above in the ltr 
embedding environment! The directionality of the embedding is the same as 
the directionality of the text immediately preceding the parentheses. I 
think this sets the reader's expectation for the display of the parens!

So for example (note: as is your convention, upper case letters designate 
RTL characters/text and lower case designate ltr):
TEXT: AS-SAYYAD AL-ALIFBAYT (w3c lead, balad1), abc (w3c lead, country2).
O.k., if the embedding directionality of this text is ltr, these parenthese 
can be displayed as ltr since there is enough ltr text both inside and outside of them.
However, if the embedding is rtl, the rtl text immediately preceding the 
parentheses makes me expect to see the parentheses displayed as rtl;
thus (sorry can't find an rtl comma on my keyboard):
I think in this case embedding directionality should determine the directionality 
of the whole maybe even of both parenthetical texts??
=> (country2 ,w3clead abc ,(balad1 ,w3clead) TYABFILA-LA DAYYAS-SA :TXET                                      
(** Thus, I think that, together with the directionality of the embedding level, 
the text that logically precedes the parens is critical to the determination of 
the directionality of the parens.)
3.2. Also, Sometimes there is no adjacent text on one side of a set of brackets, 
or on the other: although fortunately parentheses rarely begin a text block 
(except in programming, and in my writing), they often end a text block, followed 
by a neutral punctuation mark:
{ ? Have you addressed such cases in your algorithm? I must have missed something. }
* In an ltr text/embedding the above L(R)N should clearly be ltr. {And R(L)N 
should be rtl in an rtl text.} 
* In an ltr text the above should run rtl nevertheless! {And L(L)N should run 
ltr in an rtl embedding!}
These two cases above seem to me to be two obvious cases!!
However, for the next two cases the solution is not so obvious:
L(R)N in an rtl context/embedding 
* (should it remain rtl? I am unsure. Probably.)
R(L)N in an ltr context/embedding
* (same question; should it remain ltr? Probably.)

The following I am more sure of:
 L(RL)N in an rtl context/embedding
* I would make this ltr
R(RL)N in an ltr context/embedding
* I would make this rtl

4. Examples (just to give you all an idea of some uses of bidi with parens)
(I'm not 100% sure but believe that
Facebook uses any rtl??? and not 1rst strong;
thus, all text at facebook with any rtl character is processed then as rtl --
or not?
perhaps though Facebook's algorithm breaks text into parenthes but any rtl 
in or before a parenthes changes embedding direction??
Oh well.
But I don't need to discuss a particular application/site.)
In any case, below are my examples; I am assuming that the embedding is rtl 
for all cases

Example A. Embedding rtl or ltr
Sound transcription in English (EXAMPLEORDEFINITIONINOTHERLANG) Definition 
in English
* I want parens displayed ltr, with a separate embedding level for content 
inside so that it can run rtl of course as it will
=> Sound transcription in English (GNALREHTONINOITINIFEDROELPMAXE) Definition 
in English
* you resolve  as ltr only if the embedding is ltr?

Example B. Embedding rtl but should be ltr
Text EXAMPLE1INOTHERLANGUAGE (Sound transcription in English, Definition in 
** What I want:
=> Text: EGAUGNALREHTONI1ELPMAXE (Sound transcription in English, Definition 
** That is, I want parens displayed ltr again because of the opening ltr text.
* Adjacent text is all rtl
* Content of parens is strong ltr
*  you resolve the parens as ultimately -- it seems, if embedding is rtl then 
you resolve the parens as rtl? ;
rtl will not do for my content I guess but particularly not if the embedding 
is ltr (which I think it should be here with the ltr text beginning this) 
but in that case you do resolve the parens as ltr?? (I think I've read you 
In any case, I believe I can resolve the above with a line break (if a neutral 
character at the end such as a line break has the correct effect) or a declaration 
of document directionality.

Example C. Embedding ltr or rtl
English sentence (EXAMPLEINOTHERLANGUAGE, Sound transcription in English) 
More text
** Again I want parens displayed ltr in spite of the EXAMPLEINOTHERLANGUAGE; 
however in my case but not in general if the parens run rtl the the phonetic 
transcription simply appears before the Arabic text and this won't really hurt me)
* Some text in rtl some ltr in parens
 * If embedding is ltr, then your algorithm makes parens come out ltr;  this 
case is resolved corrrectly immediately.

--C.E. Whitehead

Date/Time: Wed Jul 18 13:52:15 CDT 2012
Name: Aharon Lanin
Report Type: Public Review Issue
Opt Subject: PRI #231 issue: proposed N0 violates X10 by working outside of a level run

This is a a comment on PRI #231 (the BPA). The proposal includes adding the
following rule to UAX #9:

*N0. Paired punctuation marks take the embedding direction if the enclosed
text contains a strong type of the same direction. Else, if the enclosed text
contains a strong type of the opposite direction and at least one external
neighbor also has that direction the paired punctuation marks take the
direction opposite the embedding direction. [...] This rule is applied to
those paired punctuation marks that are correctly nested and occur at the same
level without an intervening drop below their level."

This rule does not fit into the Unicode Bidirectional Algorithm as currently
formulated for a technical reason: rule X10 (which PRI #231 does not propose
to modify) states that "The remaining rules are applied to each run of
characters at the same level." The proposed N0 is one of those "remaining
rules", but attempts to apply to pairs of characters that may be separated by
a higher embedding level, and thus may be in two different level runs. This
violates X10.

Date/Time: Sun Jul 29 16:41:05 CDT 2012
Name: Matitiahu Allouche
Report Type: Public Review Issue
Opt Subject: PRI #231 (Bidi Parentheses Algorithm)

Ooops! I am past the closing date. I hope the comments
below will be considered nevertheless.

1) In the table of section 3.5, the last line of the second column R(LR)R
should be highlighted, since the UBA will resolve the open paren as LTR and
the close paren as RTL.

2) In the same table, the sixth line of the fourth column L(LR)L should be
highlighted, since the UBA will resolve the open paren as LTR and the close paren as RTL.

3) Same comment for the next line L(LR)R.

4) Line 115 mentions "the directionality of the enclosed content". It is not clear
what this directionality is when the content includes mixed LTR and RTL text.

5) For completeness, rule N0 should specify what happens when the enclosed text
is all N, even if to say that the BPA does not affect this case.

6) In section 5, example 6, I don't understand the result of the BPA. From the 
UBA display, I understand that the LTR text (Microsoft Corp) is at the logical 
end of the string. In the BPA display it appears on the starting end of the 
string. I see nothing in the definition of the BPA which should give such a result.

7) Is a solution to the current problem of mismatched parenthesis desirable?
I am not sure, because of the following reasons:
a.	The UBA is already quite complex. The BPA would add still more complexity. 
Proof is that, if my comments 1-3 and 6 above are founded, even the author of the 
proposal has missed some fine points. And if my comments are not founded, I am 
myself the one who got confused, despite the fact that I have more experience in 
bidi matters than the average person.
b.	Consider a text editor implementing the UBA and BPA by transforming the 
logical text to visual display after each keystroke. When entering an open paren,
 the BPA will not kick in, since it is not paired. When entering the closing paren, 
the BPA will kick in, possibly modifying the display of text around the opening
paren, which may be a few lines far from the typing location.
The UBA also has effects of modifying the appearance of text already entered, 
but it is always in the close neighborhood of the typing location.

8) Does the proposed solution meet expectations in terms of the naturalness of 
segmentation and directional flow of enclosed units?
The proposal assumes that opposite direction content within parentheses forms 
a unique directional run with opposite direction text on either side. When one 
of the sides has the embedding direction, I don't see that the context has opposite 
direction rather than embedded direction. In doubt, the BPA should not assume 
opposite direction for the parentheses.
Here is an example. The text in logical order (with upper case representing RTL 
letters) is 
   "I LIVE IN paris (france)."
Assuming a RTL paragraph direction, the UBA will display 
   ".(paris (france NI EVIL I"
The BPA will display
   ".paris (france) NI EVIL I"
which is better. However, since the general direction of the text is RTL, I prefer 
to have it displayed as 
   ".(france) paris NI EVIL I"
To get  this result, rule N0 can be reformulated as follows:
N0. Paired punctuation marks take the opposite direction if the enclosed text contains 
no strong type of the embedding direction and the external neighbors on both sides 
have the opposite direction. Else the paired punctuation marks take the embedding direction. 

9) Should the BPA be implemented as a new rule affecting the resolution of neutral 
types in the core UBA . proposed rule N0? 
Should the BPA, rather, be a recommended implementation using higher level protocols?
As said above, I am not sure that the BPA has more benefits than problems, but if it 
is adopted, I think it should be in the core UBA. Leaving it for a higher level 
protocol introduces one more degree of uncertainty in the behavior of the presentation 
system, and this is not something we need.

10) Are stability concerns adequately addressed?
I think that in most cases, reasonable bidi text which looks good with the UBA will 
look the same with the BPA. However here is an exception. The logical text is (where ] 
represents RLE and ^ represents PDF):
   I LIVE IN ]paris^(france).
The UBA will display it as:
   .(france)paris^] NI EVIL I
The BPA, according to the proposed N0, will assign the opposite direction to the 
parentheses, thus displaying:
   .paris^(france)] NI EVIL I
You can see that the order of "paris" and "france" is reversed. That would not happen 
with the revised N0 that I suggested in comment 8 above.

11) Are interoperability concerns during the migration period adequately addressed?
The document states: " The main stability concern therefore is that text authored using 
the BPA may display differently when rendered on a system which has not implemented the 
BPA. In such a case, the reader of that text is no worse off than they would have been 
prior to the development of the BPA."
This is not quite correct. Without the development of the BPA, the document author 
would have taken measures (like adding control characters) to create a correct display 
under the UBA. Authors writing on BPA-supporting systems are likely to create text 
which will be rendered differently on BPA-ignorant systems.
However, this is a problem which will always surface when new features are introduced.