From: Kenneth Whistler (kenw@sybase.com)
Date: Mon Nov 15 2010 - 19:43:54 CST
Asmus replied:
> On 11/15/2010 2:24 PM, Kenneth Whistler wrote:
> >> FA47 is a "compatibility character", and would have a
> >> compatibility mapping.
> > Faulty syllogism.
>
> Formally correct answer but only because of something of a design flaw
> in Unicode. When the type of mapping was decided on, people didn't fully
> expect that NFC might become widely used/enforced, making these
> distinctions appear wherever text is normalized in a distributed
> architecture.
O.k., I'm gonna have to intervene again. *hehe* Yes, there is
a design flaw here, but Asmus' explanation is also somewhat
faulty, because it flattens out the history in a way that is
liable to be misunderstood.
There is a *reason* why "when the type of mapping was decided on"
that "people didn't fully expect that NFC might become
widely used/enforced" -- but it wasn't that they were goofing
up in understanding the implications of normalization. Rather,
at that point in Unicode history NFC didn't *exist* yet, nor
had the normalization algorithm been designed.
Here, for the benefit of the standards geeks out there, are the
relevant higlights of the historical timeline involved.
June, 1992.
The canonical mappings for the CJK Compatibility characters
were *printed* (with off-by-one errors for some of them!) in
Unicode 1.0, volume 2 (= Unicode 1.0.1).
Actually, at the time, we didn't know they were "canonical"
mappings, because that concept hadn't formally been invented
yet, but the intention was clear. They were the mappings
from the "CJK compatibility ideographs" to the "real" unified
Han ideographs in the standard. The CJK compatibility characters
were all considered to be duplicates in the source standards
that didn't follow the unification rules.
July, 1996.
The formal definitions of "canonical decomposition" and
"compatibility decomposition" were first published in
Unicode 2.0. There wasn't a data file for the CJK Compatibility
Ideographs block, but the canonical mappings were *printed*
(correctly, this time) on pp. 7-470 to 7-472 of the standard.
August 4, 1998.
The first published version of UnicodeData.txt that contained
the canonical mappings for the CJK Compatibility Ideographs
was UnicodeData-2.1.5.txt for Unicode 2.1.5. (Actually,
they got into UnicodeData-2.1.4.txt on July 9, 1998, but that
wasn't a published version of the data file.)
July 23, 1999.
This was the publication data of the first approved version
of UAX #15 (Revision 15), and so is the first published definition
of NFC. (Of course UAX #15 had been in draft for some time earlier
than that, so the term "NFC" can be tracked back in the drafts
to mid-1998.)
September, 1999.
Release of Unicode 3.0 -- the first release of Unicode formally
tied to the Unicode Normalization Algorithm. (The revision
of UAX #15 for the release was actually Revision 18, dated
November 11, 1999.)
March 23, 2001.
UAX #15, Version 3.1.0. This was the version of the Unicode
Normalization Algorithm that specified the composition version
to be Version 3.1.0 and locked down normalization
forever more.
So essentially, there was a 9 year period between when the
first mappings were defined for the CJK Compatibility Ideographs
and the date beyond which it became impossible to reinterpret
or change a canonical mapping because of the lockdown of
normalization.
The problems resulting from the normalization for CJK Compatibility
Ideographs only started to become visible to people *after*
the lockdown, and when Unicode normalization started to become
a regular feature of actual processing.
And it wasn't because "people didn't fully expect that NFC might
become widely used/enforced" -- or at least not the people in
the UTC. The UAX #15 text published with Unicode 3.0 already
stated: "The W3C Character Model for the World Wide Web requires
the use of Normalization Form C for XML and related standards..."
And it wasn't because of some oversight about the canonical
mappings involving the CJK Compatibility Ideographs per se.
That same UAX #15 for Unicode 3.0 also stated: "With *all*
normalization forms singleton characters (those with singleton
canonical mappings) are replaced." So the ground facts for
the FA10 --> (NFC/NFD/NFKC/NFKD) 585C normalization pattern
were well-established and explicitly stated in 1999.
> > FA47 is a CJK Compatibility character, which means it was encoded
> > for compatibility purposes -- in this case to cover the round-trip
> > mapping needed for JIS X 0213.
> >
> > However, it has a *canonical* decomposition mapping to U+6F22.
>
> And that, of course, destroys the desired "round-trip" behavior if it is
> inadvertently applied while the data are encoded in Unicode. Hence the
> need to recreate a solution to the issue of variant forms with a
> different mechanism, the ideographic variation sequence (and
> corresponding database).
Yes, that is basically correct. But, this architectural "design flaw"
actually results from two additional requirements that accrued
to the Unicode Standard well after its initial design:
1. The requirement to be able to carry "round-trip" behavior
through distributed environments.
In the original design, the notion of how one would deal with
legacy data was conceived of primarily as a controlled and
contained conversion issue. An application/system would convert
legacy data to Unicode, and if it needed to convert back, it
could use compatibility characters for round-trip conversion.
The system would know how and when it could normalize, because
it controlled the data and the conversion.
2. The requirement to be able to maintain CJK variant glyph
distinctions in plain text data.
Again, that was not at all a part of the original Unicode
Standard design.
So the essential nature of the problem is that these new requirements
have mostly accrued to Unicode implementations *after* 2001,
more or less at the point when the lockdown of Unicode normalization
made it impossible for normalization to be adjusted in any way
to account for them.
Hence the need to construct an *alternative* approach involving
variation selectors, which would be robust and invariant under
normalization transformations.
> > The behavior in BabelPad is correct: U+6F22 is the NFC form of U+FA47.
> >
> > Easily verified, for example, by checking the FA47 entry in
> > NormalizationTest.txt in the UCD.
>
> While correct, it's something that remains a bit of a gotcha.
Yeah, well, the basic gotcha is that no matter how many times
I say it or what the Unicode Standard says, people will continue
to just assume "compatibility character" implies "compatibility
decomposition". For everybody on the list, I recommend
frequent re-reading of Section 2.3, Compatibility Characters,
of the standard:
http://www.unicode.org/versions/Unicode5.2.0/ch02.pdf
whenever somebody mentions "compatibility" in discussion
of Unicode. Yes, I suspect that people will find their
heads hurting -- but this subject *is* complex, and generalizations
that people make about "compatibility characters" are often
wrong when they don't pay attention to the details.
> Especially
> now that Unicode has charts that go to great length showing the
> different glyphs for these characters,
Well, even there the issue is complicated, because there
are CJK Compatibility Ideographs, and then there are
CJK Compatibility Ideographs. They fall into at least
3 important classes:
1. Ones which really are *unified* ideographs, despite their
names.
2. Ones which are *pronunciation* variants from KS X 1001,
and which are *not* intended to show different glyphs.
3. Ones which are *graphical* variants from other legacy
standards, and which *are* intended to show different
glyphs.
And even class 3 has subtypes, because some show variants
that are distinguished only in one legacy standard, whereas
some are themselves cross-mapped between more than one
legacy standard -- putatively because each legacy standard
shows the same variant glyph.
It is class 3 that may be adversely affected *visually* by the
application of normalization in a distributed environment.
> I would suggest adding a note to
> the charts that make clear that these distinctions are *removed* anytime
> the text is normalized, which, in a distributed architecture may happen
> anytime.
The CJK Compatibility Ideographs already have warnings attached
to them in the standard. They are repeatedly documented as "only
for round-trip compatibility with XYZ" and "They should not
be used for any other purpose."
However, I think your point is a valid one. Now that the
clear answer for maintaining legacy CJK glyph variant
distinctions in a distributed environment is via ideographic
variation sequences as registered in the IVD, it would make
sense to beef up the CJK Compatibility Ideograph documentation
with better pointers (and with accompanying rationale text)
to UTS #37 and the IVD, and to post stronger warning labels
in the code charts for CJK Compatibility Ideographs.
Perhaps someone would like to make a detailed proposal to
the UTC for how to fix the text and charts? ;-)
--Ken
This archive was generated by hypermail 2.1.5 : Mon Nov 15 2010 - 19:48:12 CST