From: Philippe Verdy (email@example.com)
Date: Tue Jul 13 2010 - 09:20:10 CDT
> De : "CE Whitehead" <firstname.lastname@example.org>
> A : email@example.com, firstname.lastname@example.org
> Copie à :
> Objet : RE: UTS#10 (collation) : French backwards level 2, and word-breakers.
> Hi, I am sort of confused; so there is no way now to put some of the weights in reverse order at the secondary level while skipping word boundaries?
> Philippe Verdy's suggestion seems reasonable, in general; however I think that not reversing the weights at word boundaries at level 2 should be simply an option for French; also I do believe that there is already a way to identify word boundaries at the primary level in the DUCET but I may be wrong -- that is the characters that define word boundaries, non-spacing characters, white space, are defined.
> So is the point then to define all word separators -- whether in the form of white space, a mandatory line break, etc. -- with a single weight in the DUCET? (Sorry to be so confused.)
There's already a standard annex covering word boundaries, and other
boundaries : lines, sentences, default grapheme clusters (including
ZWJ and ZWNJ, and the 8 Thai/Lao prepended characters, and sequences
that include double diacritics)... plus the combining sequence
boundaries that are part of the core standard for normalizations.
Word boundaries are within the simplest boundaries to compute (at
least for alphabetic scripts (this is certainly more complex for East
Asian scritps, but the same scripts, but these boundaries are not very
useful for collation purpose).
No need to reinvent the wheel specifically for UTS#10 collation, which
already needs the default grapheme cluster boundaries, as the smallest
boundaries (spanning entirely one or more combining sequences.),
possibly extended to cover multiple default grapheme clusters in
language-spacific clusters (for M-to-1 and M-to-N weight mappings).
Yes I know also that isolated combining characters may also receive
their own collation weights, because they are not necessarily combined
within M-to-1 or M-to-N weight mappings.
But the way UCA and the DUCET is built is to make sure that the result
will be consistant within at least the default grapheme clusters,
independantly of language-specific tailorings (but I'm not sure that
the UCA algorithm addresses all the consistancy issues to make sure
that this will be true for all the default grapheme clusters,
including in tailorings, when only the combining sequence boundaries
are really secured).
UTS#10 is also helping us to define the "non-default" grapheme
clusters perceived in various languages. It is really a complement to
the existing UAX for boundaries, that goes beyond just the purpose of
sorting and can be used even without considering any collation weights
and independantly of collation levels. For example these boundaries
can be used in full text indexing, in orthographic correctors, and in
semantic analysis of encoded texts, and they may also help for
enhancing the usability of text editors, or for text
selection/extraction in browsers.
As the introduction of backwards levels in UCA was made apparently
specifically for French collation, it really forgot one aspect of
French collation: that this is only wanted within single words (the
most significant secondary differences of accents are to be found at
end of each word separately, but not at end of texts of arbitrary
And even if Kenneth at Sybase thinks that this would complicate things
or slow down collation, my own experience demonstrates just the
opposite, exactly for French collation (he recognizes himself that
French UCA collation is slow, but the cause of this slowness is
because word boundaries were forgotten an algorithm that is already
considering smaller boundaries, and he also admits that a small input
buffering will be needed for correct handling of normalization,
canonical equivalence of results, and Unicode process conformance),
and it has absolutely no impact on the effective performance for
collations without backwards level (including the default
locale-neutral "root" collation directly induced from the DUCET).
This archive was generated by hypermail 2.1.5 : Tue Jul 13 2010 - 09:25:34 CDT