One of the most common things that processes implementing
the Unicode Standard need to do is compare Unicode strings.
As for ASCII or any other character encoding, Unicode
strings can simply be compared by their binary code values,
but for linguistically relevant string comparisons, more
sophisticated comparison is necessary, taking into account
casing and accents, ignoring certain characters, and so on.
The Unicode Consortium has published a separate standard devoted
specifically to this issue of string comparison, or collation:
UTS #10, Unicode Collation Algorithm (UCA).
That algorithm provides a complete specification of how
to generate collation keys for Unicode strings. Those
collation keys can then be compared directly in order to
make determinations about the comparison of the Unicode
strings they were generated from. Collation keys can
also be used in string matching and searching operations.
Committees Responsible for Collation
The Unicode Technical Committee is responsible
for the maintenance of both the Unicode Collation Algorithm
and the Default Unicode Collation Element Table (DUCET) which
provides all the basic collation key weighting information
used by the algorithm.
The CLDR Technical Committee is responsible
for maintaining information about language-specific tailoring
of the Unicode Collation Algorithm—for example, a Swedish-specific
collation, a Czech-specific collation, and so forth. Such
information is specified, using the CLDR tailoring syntax, in the
Common Locale Data Repository
Policies Regarding Collation
The UTC has defined detailed policies that it uses in the maintenance
of the DUCET table for the Unicode Collation Algorithm.
The first set of policies covers constraints on how the
existing DUCET table can be changed. Those can be found in
Change Management for the Unicode Collation
The second set of policies specifies criteria by which initial
collation weights are assigned to characters newly added to the Unicode
Standard. Those can be found in UCA Default Table Criteria for New Characters.