Re: Surrogate points

From: Kenneth Whistler (
Date: Thu Feb 03 2005 - 17:33:11 CST

  • Next message: Magda Danish \(Unicode\): "FW: Subj: Ansi/Unicode data diagnostics"

    O.k. I have been holding my tongue, but this particular
    tower of blather in the colloquy between Philippe Verdy
    and Hans Aberg requires some corrections.

    Hans Aberg said:

    > [Off the list.]
    > >> The problem with Unicode is that it seems to attempt to do to much
    > >> in this category. It should focus on the character model, and let the
    > >> other things to other work and standards. That would remove a great
    > >> deal of controversy around it.
    > >

    and Philippe Verdy responded:

    > >At least on this, I can agree with you.
    > >I think that Unicode's attempt to cover too many things in the same
    > >standard will fail in a more or less long term. The Unicode standard
    > >should be splitted into separate working domains.

    Hans and Philippe are, of course, entitled to their opinions about
    what "Unicode" should do, but this repartee seems to reflect an
    ignorance about what standardization is actually going on.

    "Unicode" is not a single standard, nor does "Unicode" *do* things.

    The *Unicode Consortium* is an SDO (Standards Development Organization).
    At current count it maintains and develops 6 different standards.
    The Unicode Standard is the largest and most important of those,
    of course, but it is only one. The Unicode Consortium is currently
    also the registration authority for two standards. It maintains the
    Common Locale Data Repository (CLDR), which depends on the LDML
    standard. (See: ) It also is the
    registration authority for ISO 15924, Script Codes.
    (See: )

    From the Bylaws of Unicode, Inc., the formal, incorporated entity
    which runs the Unicode Consortium:

    "Section 1. Purpose
     This Corporation's purpose shall be to extend, maintain and
     promote the Unicode Standard and other standards related to
     internationalization technologies."
    That wording has the approval of the Board of Directors, officers,
    and corporate members of the Unicode Consortium.

    Now, as I said, Hans and Philippe are entitled to their opinions,
    but it would be desirable if those opinions were at least grounded
    on an understanding of the actual activities of the Unicode Consortium.

    Hans responded:

    > I think this is what is causing a lot of heat on the list, a combination of
    > trying to do too much, in combination with the requirement of no change.

    Heat on the list is caused by behavior of participants on the
    list, not by the scope of the Unicode Standard, nor by stability
    requirements imposed on that standard.

    > Then, a series of issue have been resolved by compromises, which are not
    > generally useful.

    This statement is stunningly at odds with the assessment of the
    committees involved in the maintenance of the Unicode Standard
    and ISO/IEC 10646. Compromise in the face of conflicting requirements
    is the soul of consensus in the development of the standards, and
    in the case of the Unicode Standard has been responsible for the
    wide adoption and general success of the standard as an underpinning
    for text processing in worldwide IT contexts.

    > When complaints naturally arrive, one has locked ones
    > positions in view of the non-change requirement, which produces a rather
    > defensive, aggressive stance on the list.

    A standard, once widely implemented, needs to be defended against
    destabilizing changes. That should be taken as a given.

    The aggressive nature of some of the responses on the threads
    initiated by Hans isn't caused by the stability requirements
    of the standard, but rather has been responding to the
    arrogantly challenging nature of the proposals being made.

    Philippe said:

    > >Collation for example is not bound to the character encoding itself. I
    > >think it should go out of Unicode's standard,

    The UCA is not a part of the Unicode Standard, but is a separate
    standard in its own right.

    > >and be worked by another
    > >group, without being bound to Unicode character properties.

    This is a matter of opinion, of course, but I disagree with
    Philippe's assessment.

    Hans responded:

    > Other issues that should not be in Unicode are file encodings, process
    > handling.

    And they aren't dealt with by the Unicode Standard, contrary to
    Hans' implication here.

    > Also the endianess of the representation of number is languages
    > seems to be wrong.

    May "seem to be wrong" to Hans, but is not.

    > So there seems to be a range of issues that should be
    > lopped off current Unicode.

    Philippe said:

    > >I think that ISO would be a better place to work on collation, because
    > >it's not a character encoding issue, but a more general issue about
    > >handling linguistic data and semantics.

    It is certainly the case that collation is not a character encoding
    issue per se.

    But the presupposition here is that "ISO" is better equipped to handle
    matters of linguistic data and semantics. "ISO" doesn't handle anything
    of the sort. It is a Standards Development Organization that pushes
    all matters of technical expertise down into appropriate working
    groups. The subcommittee in ISO that has both the formal responsibility
    for International String Ordering now is SC2, the same subcommittee
    that deals with -- surprise -- ISO/IEC 10646. And the expertise regarding
    collation in that subcommittee resides in Working Group 2, the
    working group that does all the work on 10646.

    That alignment of activities, parallel on the ISO side for 10646 and
    14651 and on the Unicode Consortium side for the Unicode Standard
    and UCA, should hardly be surprising, because the main issue for
    both ISO/IEC 14651 and for the UCA is the appropriate extension
    of the main tables (CTT for 14651 and DUCET for UCA) as characters
    are added to the repertoire of 10646 and the Unicode Standard.

    > >A unique solution for collation
    > >will not work for all languages.

    This is true. It is also well-understood and accounted for by the
    developers and maintainers of both UCA and ISO/IEC 14651.

    > >I think that a more open standard that
    > >will be based on various profiles (including Unicode's UCA as one of
    > >those profiles) with more precise but more open definitions bound in
    > >priority to linguistic issues would be welcome.

    But this seems to reflect a lack of understanding of the nature of
    tailoring for particular collations, both in UCA and ISO/IEC 14651.

    > It has been discussed a bit in the LaTeX list, and it is clear that these
    > language and region related issues are very complex. Other issues are how to
    > represent dates in various localities, where the same language, but
    > different locality will use different conventions. For example, Australian,.
    > UK, and US conventions. Then people may make a pick between different
    > conventions in their text. So if Unicode stick its nose into those water,
    > one is likely to get water over the head.

    Please see:

    On the contrary, I'd say it is Philippe and Hans who are swimming in
    the deep end without a life preserver here.

    Philippe continued:

    > >May be
    > >could become the registry agency for those profiles (for example if the
    > >registry is made part of CLDR). But UCA and the Unicode's DUCET is
    > >unusable as such.

    This is demonstrably false. It is currently being used as the basis
    of shipping software in major implementations.

    > > New collation algorithms are needed that will make
    > >things simpler and more efficient to cover large sets of languages for
    > >which the algorithm is poor (inprecise or ambiguous) and inefficient
    > >(slow, complex to implement)

    The UCA is certainly complex to implement, but then so is every
    similar approach to multi-level, linguistically appropriate sortkey
    weighting that preceded it. A proper implementation is *NOT* slow.

    > >On the opposite, working groups on collation categorized by linguistic
    > >domains could be created at ISO, to cover several groups of languages,
    > >based only on the ISO10646 character repertoire, and with their own
    > >sets of character properties independant of Unicode, these properties
    > >becoming significant only for the covered languages.

    And that is a recipe for chaos and non-interoperability.

    It is one thing to propose that groups with regional expertise
    in sorting practice for one language or group of languages develop
    specifications regarding how that language or group of languages
    should be sorted. It is an entirely different matter to be
    proposing that they disanchor their specifications from
    the specification of Unicode character properties.

    > >another example: the set of normative properties in Unicode needed
    > >before characters can be standardized is becoming too large. This is a
    > >critical issue, because it is slowing the standardization process also
    > >at ISO10646.


    > >So Unicode tends to assign too early normative properties
    > >that will become unusable later, and that will require application
    > >developers to use their own non-compliant solutions (examples found and
    > >needed today with Biblic Hebrew

    This is an invalid generalization from a set of known issues regarding
    fixed position combining classes for Hebrew points -- known issues
    that have been chewed over ad nauseum on the Hebrew list here, and
    which have yielded to solutions as a result of the very process
    of compromise apparently decried by Hans above.

    > >and Phoenician).

    And that is an utter non-sequitur, because the normative properties
    of Phoenician characters do not and never have had any connection to the
    controversy over the encoding of the script.

    Hans continued:

    > That seems to be problem with Unicode: by wanting to do too much, one will
    > provide norms that merely will be disobeyed. THis is a gener problem with
    > standards, not only Unicode. Therefore, quite few standrds will never be in
    > effect used.

    Bypassing the faulty logic here, I would point out that the Unicode
    Standard *is* used and its specifications *are* followed, rather well,
    in fact, by many vendors.

    Philippe continued:
    > >Splitting the standard would help abandonning some parts of it in favor
    > >of other ones. So applications could be still conforming to ISO10646
    > >and a reduced Unicode standard, but could adopt other standards for all
    > >other domains not covered by the core Unicode standard.

    This is utter pie in the sky. Not only do I see no motivation for this --
    there is nobody waiting in the wings to take on this task.

    > >doing this
    > >should not require reencoding texts. But it could really accelerate the
    > >production of documents with still unencoded scripts or characters.

    And this is nonsense. It would only increase uncertainty and
    confusion, and would *slow down* the production of documents for
    still unencoded scripts or characters.

    Experts representing living minority scripts not yet in the Unicode
    Standard or not yet fully covered by the Unicode Standard "get it"
    now. In the past year there have been excellent examples of
    cooperative collaboration that have speeded the encoding for
    Tifinagh, N'Ko (for Mandekan speakers in Guinea and neighboring
    countries), extensions for Ethiopic, and Balinese. Many others are
    in the works.

    It is a shame that Philippe doesn't "get it" that splitting such
    efforts off from the general process of extension of the Unicode
    Standard would have the net effect of isolating and disenfranchising
    such groups, rather than enabling them in the IT world.

    Hans said:

    > I think that Unicode should focus on providing the character set, the
    > character numbering, and in some cases, rules for combined characters.

    Hans is entitled to think that, but he is wrong. The accumulated
    engineering expertise of the software engineers working on
    the standard and its implementation over the last 15 years is,
    in fact, what has driven the Unicode Consortium to incorporate
    all kinds of semantic information beyond mere character
    encoding repertoire into the Unicode Standard. Han's position
    is approximately where the Unicode founders were at in 1989, in
    their thinking about what the task was for the Unicode Standard.
    He has a little catching up to do here.

    > If
    > the encoding issue would have been handled correctly, it would have been
    > completely independent of these issues.

    It isn't clear what Hans means by that statement, but whatever
    it is, I suspect he's wrong on that, too. ;-)

    Philippe continued:

    > >Finally, Unicode does not cover a domain which is very important for
    > >the creation of numeric text corpus: orthographies (and their
    > >associated conventions).

    That is true.

    > >This is a place where nothing can be
    > >standardized without breaking some Unicode conformance level,

    But that statement is clearly false.

    > >even
    > >though standard orthographies could be much more easily developed based
    > >only on the ISO10646 repertoire definition.

    And so is that one.

    Hans responded:
    > This clearly belongs to the question of lexing and parsing a sequence of
    > characters. Unicode should stay out of that as much as possible, I think.

    If by "this", Hans is referring to the development of orthographies
    as mentioned by Philippe, then that has little to do with lexing
    and parsing issues, per se. But I agree that the Unicode Standard
    should not (and does not) specify anything regarding orthographies.
    > >So the good question is: are all those character properties in Unicode
    > >needed or pertinent to cover all languages of the world? Unicode has no
    > >right and no expertise in linguistic issues; only in encoding issues.

    First of all, Unicode character properties have no direct bearing
    on linguistic issues, anyway.

    Second, the developers of the Unicode Standard *do* have expertise
    in linguistic issues, contrary to Philippe's apparent claim. The
    Unicode Standard itself is not, of course, directed at standardizing
    any linguistic or orthographic issue.

    Hans started freewheeling:
    > I can think of more than one character set, going in different directions
    > relative Unicode: One that is optimized by having as few characters as
    > possible. Another, going the opposite direction, might be more ample in the
    > set of characters, perhaps having one for each language-locality combination
    > that is unique. I do not think there is one set that is the right one; it
    > depends on what design objectives on has.

    Well, yes, I can think of things like that, too. And the point is?

    What we have for a *universal* character encoding standard is the
    Unicode Standard (and the synchronized standard ISO/IEC 10646).
    That is the result of the combined efforts of literally 100's of
    contributors over a 15-year time span.

    Perhaps Hans would be so kind as to gift us with particulars as to
    by whom and how his imagined alternatives are to be realized and then
    rolled out into real implementations?

    This archive was generated by hypermail 2.1.5 : Thu Feb 03 2005 - 17:35:59 CST