[Unicode]  Technical Reports
 

Unicode Technical Standard #35

Unicode Locale Data Markup Language (LDML)
Part 5: Collation

Version 24
Editors CLDR committee members
Date 2013-09-18
This Version http://www.unicode.org/reports/tr35/tr35-33/tr35.html
Previous Version http://www.unicode.org/reports/tr35/tr35-31/tr35.html
Latest Version http://www.unicode.org/reports/tr35/
Corrigenda http://unicode.org/cldr/corrigenda.html
Latest Proposed Update http://www.unicode.org/reports/tr35/proposed.html
Namespace http://cldr.unicode.org/
DTDs http://unicode.org/cldr/dtd/24/
Revision 33

Summary

This document describes parts of an XML format (vocabulary) for the exchange of structured locale data. This format is used in the Unicode Common Locale Data Repository.

This is a partial document, describing only those parts of the LDML that are relevant for collation (sorting, searching & grouping). For the other parts of the LDML see the main LDML document and the links above.

Status

This document has been reviewed by Unicode members and other interested parties, and has been approved for publication by the Unicode Consortium. This is a stable document and may be used as reference material or cited as a normative reference by other specifications.

A Unicode Technical Standard (UTS) is an independent specification. Conformance to the Unicode Standard does not imply conformance to any UTS.

Please submit corrigenda and other comments with the CLDR bug reporting form [Bugs]. Related information that is useful in understanding this document is found in the References. For the latest version of the Unicode Standard see [Unicode]. For a list of current Unicode Technical Reports see [Reports]. For more information about versions of the Unicode Standard, see [Versions].

Parts

The LDML specification is divided into the following parts:

Contents of Part 5, Collation

1 CLDR Collation

Collation is the general term for the process and function of determining the sorting order of strings of characters, for example for lists of strings presented to users, or in databases for sorting and selecting records.

Collation varies by language, by application (some languages use special phonebook sorting), and other criteria (for example, phonetic vs. visual).

CLDR provides collation data for many languages and styles. The data supports not only sorting but also language-sensitive searching and grouping under index headers. All CLDR collations are based on the [UCA] default order, with common modifications applied in the CLDR root collation, and further tailored for language and style as needed.

1.1 CLDR Collation Algorithm

The CLDR collation algorithm is an extension of the Unicode Collation Algorithm.

U+FFFE

U+FFFE maps to a CE with special minimal weights on all levels, including case, quaternary and identical levels — which may require special code for those levels. Its primary weight is not "variable": U+FFFE must not become ignorable in alternate handling. This allows for Merging Sort Keys within code point space. For example, when sorting names in a database, a sortable string can be formed with last_name + '\uFFFE' + first_name. These strings would sort properly, without ever comparing the last part of a last name with the first part of another first name.

For backwards secondary level sorting, text segments separated by U+FFFE are processed in forward segment order, and within each segment the secondary weights are compared backwards. This is so that such combined strings are processed consistently with concatenating their sort keys.

Context-Sensitive Mappings

Contraction matching, as in the UCA, starts from the first character of the contraction string. It slows down processing of that first character even when none of its contractions matches. In some cases, it is preferrable to change such contractions to mappings with a prefix (context before a character), so that complex processing is done only when the less-frequently occurring trailing character is encountered.

For example, the DUCET contains contractions for several variants of L· (L followed by middle dot). Collating ASCII text is slowed down by contraction matching starting with L/l. In the CLDR root collation, these contractions are replaced by prefix mappings (L|·) which are triggered only when the middle dot is encountered. CLDR also uses prefix rules in the Japanese tailoring, for processing of Hiragana/Katakana length and iteration marks.

The mapping is conditional on the prefix match but does not change the mappings for the preceding text. As a result, a contraction mapping for "px" can be replaced by a prefix rule "p|x" only if px maps to the collation elements for p followed by the collation elements for "x if after p". In the DUCET, L· maps to CE(L) followed by a special secondary CE (which differs from CE(·) when · is not preceded by L). In the CLDR root collation, L has no context-sensitive mappings, but · maps to that special secondary CE if preceded by L.

A prefix mapping for p|x behaves mostly like the contraction px, except when there is a contraction that overlaps with the prefix, for example one for "op". A contraction matches only new text (and consumes it), while a prefix matches only already-consumed text.

Note: Matching of discontiguous contractions should be implemented without rewriting the text (unlike in the [UCA] algorithm specification), so that prefix matching is predictable. (It should also help with contraction matching performance.) An implementation that does rewrite the text, as in the UCA, will get different results for some (unusual) combinations of contractions, prefix rules, and input text.

Prefix matching uses a simple longest-match algorithm (op|c wins over p|c). It is recommended that prefix rules be limited to mappings where both the prefix string and the mapped string begin with an NFC boundary (that is, with a normalization starter that does not combine backwards). (In op|ch both o and c should be starters (ccc=0) and NFC_QC=Yes.) Otherwise, prefix matching would be affected by canonical reordering and discontiguous matching, like contractions. Prefix matching is thus always contiguous.

A character can have mappings with both prefixes (context before) and contraction suffixes. Prefixes are matched first. This is to keep them reasonably implementable: When there is a mapping with both a prefix and a contraction suffix (like in Japanese: ぐ|ゞ), then the matching needs to go in both directions. The contraction might involve discontiguous matching, which needs complex text iteration and handling of skipped combining marks, and will consume the matching suffix. Prefix matching should be first because, regardless of whether there is a match, the implementation will always return to the original text index (right after the prefix) from where it will start to look at all of the contractions for that prefix.

If there is a mapping for p|c where c is a single character, and text "...pc...", then the p|c mapping should "win" over any contractions that start with c but do not have the prefix.

Consider the following mappings.

  1. p → CE(p)
  2. h → CE(h)
  3. c → CE(c)
  4. ch → CE(d)
  5. p|c → CE(u)
  6. p|ci → CE(v)

With these, text collates like this:

However, if the mapping p|c → CE(u) is missing, then text "pch" maps to CE(p)CE(d).

2 Root Collation

The CLDR root collation order is based on the Default Unicode Collation Element Table (DUCET) defined in UTS #10: Unicode Collation Algorithm [UCA]. It is used by all other locales by default, or as the base for their tailorings. (For a chart view of the UCA, see Collation Chart [UCAChart].)

Starting with CLDR 1.9, CLDR uses modified tables for the root collation order. The root locale ordering is tailored in the following ways:

2.1 Grouping classes of characters

As of Version 6.1.0, the DUCET puts characters into the following ordering:

(There are a few exceptions to this general ordering.)

The CLDR root locale modifies the DUCET tailoring by ordering the common characters more strictly by category:

What the regrouping allows is for users to parametrically reorder the groups. For example, users can reorder numbers after all scripts, or reorder Greek before Latin.

The relative order within each of these groups still matches the DUCET. Symbols, punctuation, and numbers that are grouped with a particular script stay with that script. The differences between CLDR and the DUCET order are:

  1. CLDR groups the numbers together after currency symbols, instead of splitting them with some before and some after. Thus the following are put after currencies and just before all the other numbers.

    U+09F4 ( ৴ ) [No] BENGALI CURRENCY NUMERATOR ONE
    ...
    U+1D371 ( 𝍱 ) [No] COUNTING ROD TENS DIGIT NINE

  2. CLDR handles a few other characters differently
    1. U+10A7F ( 𐩿 ) [Po] OLD SOUTH ARABIAN NUMERIC INDICATOR is put with punctuation, not symbols
    2. U+20A8 ( ₨ ) [Sc] RUPEE SIGN and U+FDFC ( ﷼ ) [Sc] RIAL SIGN are put with currency signs, not with R and REH.

2.2 Non-variable symbols

There are multiple Variable-Weighting options in the UCA for symbols and punctuation, including non-ignorable and shifted. With the shifted option, almost all symbols and punctuation are ignored—except at a fourth level. The CLDR root locale ordering is modified so that symbols are not affected by the shifted option. That is, by default, symbols are not “variable” in CLDR. So shifted only causes whitespace and punctuation to be ignored, but not symbols (like ♥). The DUCET behavior can be specified with a locale ID using the "vt" keyword, to set the Variable section to include all of the symbols below it, or be set parametrically where implementations allow access.

See also:

2.3 Additional contractions for Tibetan

Ten contractions are added for Tibetan: Two to fulfill well-formedness condition 5, and eight more to preserve the default order for Tibetan. For details see UTS #10, Section 3.8.2, Well-Formedness of the DUCET.

2.4 Tailored noncharacter weights

U+FFFE and U+FFFF have special tailorings:

U+FFFF: This code point is tailored to have a primary weight higher than all other characters. This allows the reliable specification of a range, such as “Sch” ≤ X ≤ “Sch\uFFFF”, to include all strings starting with "sch" or equivalent.

U+FFFE: This code point produces a CE with special minimal weights on all levels. For details see the CLDR Collation Algorithm above.

UCA (beginning with version 6.3) also maps U+FFFD to a special collation element with a very high primary weight, so that it is reliably non-variable, for use with ill-formed code unit sequences.

In CLDR, so as to maintain the special collation elements, U+FFFD..U+FFFF are not further tailorable, and nothing can tailor to them. That is, neither can occur in a collation rule. For example, the following rules are illegal:

&\uFFFF < x

&x <\uFFFF

Note:

2.5 Root Collation Data Files

The CLDR root collation data files are in the CLDR repository and release, under the path common/uca/.

For most data files there are _SHORT versions available. They contain the same data but only minimal comments, to reduce the file sizes.

Comments with DUCET-style weights in files other than allkeys_CLDR.txt and allkeys_DUCET.txt use the weights defined in allkeys_CLDR.txt.

2.6 Root Collation Data File Formats

The file formats may change between versions of CLDR. The formats for CLDR 23 and beyond are as follows. As usual, text after a # is a comment.

allkeys_CLDR.txt

This file defines CLDR’s tailoring of the DUCET, as described in Section 2, Root Collation .

The format is similar to that of allkeys.txt, although there may be some differences in whitespace.

FractionalUCA.txt

The format is illustrated by the following sample lines, with commentary afterwards.

[UCA version = 6.0.0]

Provides the version number of the UCA table.

[Unified_Ideograph 4E00..9FCC FA0E..FA0F FA11 FA13..FA14 FA1F FA21 FA23..FA24 FA27..FA29 3400..4DB5 20000..2A6D6 2A700..2B734 2B740..2B81D]

Lists the ranges of Unified_Ideograph characters in collation order. They map to collation elements with implicit (constructed) primary weights.

0000; [,,]     # Zyyy Cc       [0000.0000.0000]        * <NULL>

Provides a weight line. The first element (before the ";") is a hex codepoint sequence. The second field is a sequence of collation elements. Each collation element has 3 parts separated by commas: the primary weight, secondary weight, and tertiary weight. The tertiary weight actually consists of two components: the top two bits (0xC0) are used for the case level, and should be masked off where a case level is not used.

A weight is either empty (meaning a zero or ignorable weight) or is a sequence of one or more bytes. The bytes are interpreted as a "fraction", meaning that the ordering is 04 < 05 05 < 06. The weights are constructed so that no weight is an initial subsequence of another: that is, having both the weights 05 and 05 05 is illegal. The above line consists of all ignorable weights.

The vertical bar (“|”) character is used to indicate context, as in:

006C | 00B7; [, DB A9, 05]
This example indicates that if U+00B7 appears immediately after U+006C, it is given the corresponding collation element instead. This syntax is roughly equivalent to the following contraction, but is more efficient. For details see the specification of Context-Sensitive Mappings above.
006C 00B7; CE(006C) [, DB A9, 05]

Single-byte primary weights are given to particularly frequent characters, such as space, digits, and a-z. More frequent characters are given two-byte weights, while relatively infrequent characters are given three-byte weights. For example:

...
0009; [03 05, 05, 05] # Zyyy Cc       [0100.0020.0002]        * <CHARACTER TABULATION>
...
1B60; [06 14 0C, 05, 05]    # Bali Po       [0111.0020.0002]        * BALINESE PAMENENG
...
0031; [14, 05, 05]    # Zyyy Nd       [149B.0020.0002]        * DIGIT ONE

The assignment of 2 vs 3 bytes does not reflect importance, or exact frequency.

# SPECIAL MAX/MIN COLLATION ELEMENTS
FFFE; [02, 02, 02]     # Special LOWEST primary, for merge/interleaving
FFFF; [EF FE, 05, 05]  # Special HIGHEST primary, for ranges

The two tailored noncharacters have their own weights.

F967; [U+4E0D]  # Hani Lo       [FB40.0020.0002][CE0D.0000.0000]        * CJK COMPATIBILITY IDEOGRAPH-F967
2F02; [U+4E36, 10]      # Hani So       [FB40.0020.0004][CE36.0000.0000]        * KANGXI RADICAL DOT
2E80; [U+4E36, 70, 20]  # Hani So       [FB40.0020.0004][CE36.0000.0000][0000.00FC.0004]        * CJK RADICAL REPEAT

Some collation elements are specified by reference to other mappings. This is particularly useful for Han characters which are given implicit/constructed primary weights; the reference to a Unified_Ideograph makes these mappings independent of implementation details. This technique may also be used in other mappings to show the relationship of character variants.

The referenced character must have a mapping listed earlier in the file, or the mapping must have been defined via the [Unified_Ideograph] data line. The referenced character must map to exactly one collation element.

[U+4E0D] copies U+4E0D’s entire collation element. [U+4E36, 10] copies U+4E36’s primary and secondary weights and specifies a different tertiary weight. [U+4E36, 70, 20] only copies U+4E36’s primary weight and specifies other secondary and tertiary weights.

FDD1 00A0;      [03 02 02, 05, 05]      # SPACE first primary starts reordering group
FDD1 03A9;      [5D 04 02, 05, 05]      # GREEK first primary starts reordering group (compressible)
FDD1 03E2;      [5D 64 02, 05, 05]      # COPTIC first primary

These are special mappings with primaries at the boundaries of scripts and reordering groups. They serve as tailoring boundaries, so that tailoring near the first or last character of a script or group places the tailored item into the same group. Beginning with CLDR 24, each of these is a contraction of U+FDD1 with the script sample character, mapping to the first possible primary weight per script or group. They can be enumerated for implementations of Collation Indexes. (Earlier versions mapped contractions with U+FDD0 to the last primary weights of each group but not each script.)

FDD0 0034;      [11, 05, 05]    # lead byte for numeric sorting

This mapping specifies the lead byte for numeric sorting. It must be different from the lead byte of any other primary weight, otherwise numeric sorting would generate ill-formed collation elements. Therefore, this mapping itself must be excluded from the set of regular mappings. This value can be ignored by implementations that do not support numeric sorting. (Other contractions with U+FDD0 can normally be ignored altogether.)

# HOMELESS COLLATION ELEMENTS
FDD0 0063; [, 97, 3D]       # [15E4.0020.0004] [1844.0020.0004] [0000.0041.001F]    * U+01C6 LATIN SMALL LETTER DZ WITH CARON
FDD0 0064; [, A7, 09]       # [15D1.0020.0004] [0000.0056.0004]     * U+1DD7 COMBINING LATIN SMALL LETTER C CEDILLA
FDD0 0065; [, B1, 09]       # [1644.0020.0004] [0000.0061.0004]     * U+A7A1 LATIN SMALL LETTER G WITH OBLIQUE STROKE

The DUCET has some weights that don't correspond directly to a character. To allow for implementations to have a mapping for each collation element (necessary for certain implementations of tailoring), this requires the construction of special sequences for those weights. These collation elements can normally be ignored.

Next, a number of tables are defined. The function of each of the tables is summarized afterwards.

# VALUES BASED ON UCA
...
[first regular [0D 0A, 05, 05]] # U+0060 GRAVE ACCENT
[last regular [7A FE, 05, 05]] # U+1342E EGYPTIAN HIEROGLYPH AA032
[first implicit [E0 04 06, 05, 05]] # CONSTRUCTED
[last implicit [E4 DF 7E 20, 05, 05]] # CONSTRUCTED
[first trailing [E5, 05, 05]] # CONSTRUCTED
[last trailing [E5, 05, 05]] # CONSTRUCTED
...

This table summarizes ranges of important groups of characters for implementations.

# Top Byte => Reordering Tokens
[top_byte     00      TERMINATOR ]    #       [0]     TERMINATOR=1
[top_byte     01      LEVEL-SEPARATOR ]       #       [0]     LEVEL-SEPARATOR=1
[top_byte     02      FIELD-SEPARATOR ]       #       [0]     FIELD-SEPARATOR=1
[top_byte     03      SPACE ] #       [9]     SPACE=1 Cc=6 Zl=1 Zp=1 Zs=1
...

This table defines the reordering groups, for script reordering. The table maps from the first bytes of the fractional weights to a reordering token. The format is "[top_byte " byte-value reordering-token "COMPRESS"? "]". The "COMPRESS" value is present when there is only one byte in the reordering token, and primary-weight compression can be applied. Most reordering tokens are script values; others are special-purpose values, such as PUNCTUATION. Beginning with CLDR 24, this table precedes the regular mappings, so that parsers can use this information while processing and optimizing mappings.

# Reordering Tokens => Top Bytes
[reorderingTokens     Arab    61=910 62=910 ]
[reorderingTokens     Armi    7A=22 ]
[reorderingTokens     Armn    5F=82 ]
[reorderingTokens     Avst    7A=54 ]
...

This table is an inverse mapping from reordering token to top byte(s). In terms like "61=910", the first value is the top byte, while the second is informational, indicating the number of primaries assigned with that top byte.

# General Categories => Top Byte
[categories   Cc      03{SPACE}=6 ]
[categories   Cf      77{Khmr Tale Talu Lana Cham Bali Java Mong Olck Cher Cans Ogam Runr Orkh Vaii Bamu}=2 ]
[categories   Lm      0D{SYMBOL}=25 0E{SYMBOL}=22 27{Latn}=12 28{Latn}=12 29{Latn}=12 2A{Latn}=12...

This table is informational, providing the top bytes, scripts, and primaries associated with each general category value.

# FIXED VALUES
[fixed first implicit byte E0]
[fixed last implicit byte E4]
[fixed first trail byte E5]
[fixed last trail byte EF]
[fixed first special byte F0]
[fixed last special byte FF]

[fixed secondary common byte 05]
[fixed last secondary common byte 45]
[fixed first ignorable secondary byte 80]

[fixed tertiary common byte 05]
[fixed first ignorable tertiary byte 3C]
		

The final table gives certain hard-coded byte values. The "trail" area is provided for implementation of the "trailing weights" as described in the UCA.

UCA_Rules.txt

The format for this file uses the CLDR collation syntax, see Section 3, Collation Tailorings.

3 Collation Tailorings

<!ELEMENT collations (alias | (defaultCollation?, collation*, special*)) >

<!ELEMENT defaultCollation ( #PCDATA ) >

This element of the LDML format contains one or more collation elements, distinguished by type. Each collation contains elements with parametric settings, or rules that specify a certain sort order, as a tailoring of the root order, or both.

Each locale may have multiple sort orders. The defaultCollation element defines the default tailoring for a locale and its sublocales. For example:

To allow implementations in reduced memory environments to use CJK sorting, there are also short forms of each of these collation sequences. These provide for the most common characters in common use, and are marked with alt="short".

Note:

3.1 Version

The version attribute is used in case a specific version of the UCA is to be specified. It is optional, and is specified if the results are to be identical on different systems. If it is not supplied, then the version is assumed to be the same as the Unicode version for the system as a whole. In general, tailorings should be defined so as to minimize dependence on the underlying UCA version, by explicitly specifying the behavior of all characters used to write the language in question.

Note: For version 3.1.1 of the UCA, the version of Unicode must also be specified with any versioning information; an example would be "3.1.1/3.2" for version 3.1.1 of the UCA, for version 3.2 of Unicode. This was changed by decision of the UTC, so that dual versions were no longer necessary. So for UCA 4.0 and beyond, the version just has a single number.

3.2 Collation Element

<!ELEMENT collation (alias | ( import*, settings?, suppress_contractions?, optimize?, cr*, special*)) >

The tailoring syntax is designed to be independent of the actual weights used in any particular UCA table. That way the same rules can be applied to UCA versions over time, even if the underlying weights change. The following illustrates the overall structure of a collation:

<collation>
  <settings caseLevel="on"/>
  <cr><![CDATA[
    &c < k
  ]]></cr>
</collation>

3.3 Setting Options

The following parametric settings are attributes of <settings>. For example, <settings strength="secondary"> will only compare strings based on their primary and secondary weights. In rule syntax, these are of the form [keyword value].

If the attribute is not present, the CLDR default (or the default for the locale, if there is one) is used. That default is listed in bold italics. Where there is a UCA default that is different, it is listed in bold with (UCA default). Note that the default value for a locale may be different than the default value for the attribute, so the defaults here are not defaults for the corresponding keywords.

The Example cells include an LDML example followed by the same example in basic syntax settings attributes as well as rule syntax. In LDML files, parametric settings should be specified via settings attributes, rather than in the rules.

Collation Settings
BCP47 Key Attribute BCP47 Value Options Example  Description
ks strength level1 primary (1)
strength = "primary"

[strength 1]
Sets the default strength for comparison, as described in the [UCA]. Note that strength setting of greater than 4 may have the same effect as identical, depending on the locale and implementation.
level2 secondary (2)
level3 tertiary (3)
level4 quaternary (4)
identic identical (5)
ka alternate noignore non-ignorable
alternate = "non-ignorable"

[alternate non-ignorable]
Sets alternate handling for variable weights, as described in [UCA], where "shifted" causes certain characters to be ignored in comparison. The default for LDML is different than it is in the UCA. In LDML, the default for alternate handling is non-ignorable, while in UCA it is shifted. In addition, in LDML only whitespace and punctuation are variable.
shifted shifted (UCA default)
n/a blanked
kb backwards true on backwards = "on"

[backwards 2]
Sets the comparison for the second level to be backwards, as described in [UCA].
false off
kk normalization true on (UCA default) normalization = "off"

[normalization off]
If on, then the normal [UCA] algorithm is used. If off, then all strings that are in [FCD] and do not contain any composite combining marks will sort correctly, but others will not necessarily sort correctly.
So should only be set off if the the strings to be compared are in FCD and do not contain any composite combining marks. Composite combining marks are: { U+0344, U+0F73, U+0F75, U+0F81 } [[:^lccc=0:]&[:toNFD=/../:]] (These characters must be decomposed for discontiguous contractions to work properly. Use of these characters is discouraged by the Unicode Standard.)
Note that the default for CLDR locales may be different than in the UCA. The rules for particular locales have it set to on: those locales whose exemplar characters (in forms commonly interchanged) would be affected by normalization.
false off
kc caseLevel true on caseLevel = "off"

[caseLevel on]
If set to on, a level consisting only of case characteristics will be inserted in front of tertiary level, as a "Level 2.5". To ignore accents but take case into account, set strength to primary and case level to on. For details, see Section 3.13, Case Parameters .
false off
kf caseFirst upper upper caseFirst = "off"

[caseFirst off]
If set to upper, causes upper case to sort before lower case. If set to lower, causes lower case to sort before upper case. Useful for locales that have already supported ordering but require different order of cases. Affects case and tertiary levels. For details, see Section 3.13, Case Parameters .
lower lower
false off
kh hiraganaQuaternary true on hiragana­Quaternary = "on"

[hiraganaQ on]
Controls special treatment of Hiragana code points on quaternary level. If turned on, Hiragana codepoints will get lower values than all the other non-variable code points in shifted. That is, the normal Level 4 value for a regular collation element is FFFF, as described in [UCA], Section 3.6, Variable Weighting. This is changed to FFFE for [:script=Hiragana:] characters. The strength must be greater or equal than quaternary if this attribute is to have any effect.
false off
kn numeric true on numeric = "on"

[numericOrdering on]
If set to on, any sequence of Decimal Digits (General_Category = Nd in the [UAX44]) is sorted at a primary level with its numeric value. For example, "A-21" < "A-123". The computed primary weights are all at the start of the digit reordering group. Thus with an untailored UCA table, "a$" < "a0" < "a2" < "a12" < "a⓪" < "aa".
false off
vt variableTop See Appendix Q: Locale Extension Keys and Types. uXXuYYYY

(the default is set to the highest punctuation, thus including spaces and punctuation, but not symbols)
variableTop = "uXXuYYYY"

& \u00XX\uYYYY < [variable top]

The Option value is an encoded Unicode string, with code points in hex, leading zeros removed, and 'u' inserted between successive elements.

The BCP47 value is described in Appendix Q: Locale Extension Keys and Types.

Sets the string value for the variable top. All the code points with primary strengths less than or equal to that string will be considered variable, and thus affected by the alternate handling. Variables are ignorable by default in [UCA], but not in CLDR. See below for more information.

kr reorder a sequence of one or more reorder codes: space, punct, symbol, currency, digit, or any BCP47 script ID reorder = "Grek digit"

[reorder Grek digit]
Specifies a reordering of scripts or other significant blocks of characters such as symbols, punctuation, and digits. For the precise meaning and usage of the reorder codes, see Section 3.12, Collation Reordering.
n/a match-boundaries: n/a none
match-boundaries = "whole-word"

n/a
Defined by Section 8, Searching and Matching of [UCA].
n/a whole-character
n/a whole-word
n/a match-style n/a minimal
match-style = "medial"

n/a
Defined by Section 8, Searching and Matching of [UCA].
n/a medial
n/a maximal

Variable Top (vt) bears more explanation. Users may want to include more or fewer characters as Variable. For example, someone could want to restrict the Variable characters to just include space marks. In that case, variableTop would be set to U+1680 (see UCA Variable chart). Alternatively, someone could want more of the Common characters in them, and include characters up to (but not including '0'), by setting variableTop to be U+20BA (in Unicode 6.2; see UCA Common chart).

The effect of these settings is to customize to ignore different sets of characters when comparing strings. For example, the locale identifier "de-u-ka-shifted-vt-0024" is requesting settings appropriate for German, including German sorting conventions, and that '$' and characters sorting below it are ignored in sorting.

3.4 Collation Rule Syntax

<!ELEMENT cr #PCDATA >

The goal for the collation rule syntax is to have clearly expressed rules with a concise format. The CLDR rule syntax is a subset of the [ICUCollation] syntax.

For the CLDR root collation, the FractionalUCA.txt file defines all mappings for all of Unicode directly, and it also provides information about script boundaries, reordering groups, and other details. For tailorings, this is neither necessary nor practical. In particular, while the root collation sort order rarely changes for existing characters, their numeric collation weights change with every version. If tailorings also specified numeric weights directly, then they would have to change with every version, parallel with the root collation. Instead, for tailorings, mappings are added and modified relative to the root collation. (There is no syntax to remove mappings, except via special <suppress_contractions>.)

The ASCII [:P:] and [:S:] characters are reserved for collation syntax: [\u0021-\u002F \u003A-\u0040 \u005B-\u0060 \u007B-\u007E]

Unicode Pattern_White_Space characters between tokens are ignored. Unquoted white space terminates reset and relation strings.

A pair of ASCII apostrophes encloses quoted literal text. They are normally used to enclose a syntax character or white space, or a whole reset/relation string containing one or more such characters, so that those are parsed as part of the reset/relation strings rather than treated as syntax. A pair of immediately adjacent apostrophes is used to encode one apostrophe.

Code points can be escaped with \uhhhh and \U00hhhhhh escapes, as well as common escapes like \t and \n . (For details see the documentation of ICU UnicodeString::unescape().) This is particularly useful for default-ignorable code points, combining marks, visually indistinct variants, hard-to-type characters, etc. These sequences are unescaped before the rules are parsed; this means that even escaped syntax and white space characters need to be enclosed in apostrophes. For example: &'\u0020'='\u3000'

The ASCII double quote must be both escaped (so that the collation syntax can be enclosed in pairs of double quotes in programming environments) and quoted. For example: &'\u0022'<<<x

Comments are allowed at the beginning, and after any complete reset, relation, setting, or command. A comment begins with a # and extends to the end of the line (according to the Unicode Newline Guidelines).

The collation syntax is case-sensitive.

3.5 Orderings

The root collation mappings form the initial state. Mappings are added and removed via a sequence of rule chains. Each tailoring rule builds on the current state after all of the preceding rules (and is not affected by any following rules). Rule chains may alternate with comments, settings, and special commands.

A rule chain consists of a reset followed by one or more relations. The reset position is a string which maps to one or more collation elements according to the current state. A relation consists of an operator and a string; it maps the string to the current collation elements, modified according to the operator.

Specifying Collation Ordering
Relation Operator Basic Example Description
& & Z Map Z to collation elements according to the current state. These will be modified according to the following relation operators and then assigned to the corresponding relation strings.
< & a
< b
Make 'b' sort after 'a', as a primary (base-character) difference
<< & a
<< ä
Make 'ä' sort after 'a' as a secondary (accent) difference
<<< & a
<<< A
Make 'A' sort after 'a' as a tertiary (case/variant) difference
<<<< & か
<<<< カ
Make 'カ' (Katakana Ka) sort after 'か' (Hiragana Ka) as a quaternary difference
& v
= w 
Make 'w' sort identically to 'v'

The following shows the result of serially applying three rules.

  Rules Result Comment
1 & a < g ... a <1 g ... Put g after a.
2 & a < h < k ... a <1 h <1 k <1 g ... Now put h and k after a (inserting before the g).
3 & h << g ... a <1 h <1 g <1 k ... Now put g after h (inserting before k).

Notice that relation strings can occur multiple times, and thus override previous rules.

Each relation uses and modifies the collation elements of the immediately preceding reset position or relation. A rule chain with two or more relations is equivalent to a sequence of “atomic rules” where each rule chain has exactly one relation, and each relation is followed by a reset to this same relation string.

Example:

Rules Equivalent Atomic Rules
& b < q <<< Q
& a < x <<< X << q <<< Q < z
& b < q
& q <<< Q
& a < x
& x <<< X
& X << q
& q <<< Q
& Q < z

This is not always possible because prefix and extension strings can occur in a relation but not in a reset (see below).

The relation operator = maps its relation string to the current collation elements. Any other relation operator modifies the current collation elements as follows.

In all cases, even for = , the case bits are recomputed according to Section 3.13, Case Parameters. (This can be skipped if an implementation does not support the caseLevel or caseFirst settings.)

For example, &ae<x maps ‘x’ to two collation elements. The first one is the same as for ‘a’, and the second one has a primary weight between those for ‘e’ and ‘f’. As a result, ‘x’ sorts between “ae” and “af”. (If the primary of the first collation element was incremented instead, then ‘x’ would sort after “az”. While also sorting primary-after “ae” this would be surprising and sub-optimal.)

Some additional operators are provided to save space with large tailorings. The addition of a * to the relation operator indicates that each of the following single characters are to be handled as if they were separate relations with the corresponding strength. In the basic syntax, these are expressed by adding a * to the operation.

Abbreviating Ordering Specifications
Relation Operator Example Equivalent
<* & a
<* bcd 
& a
< b < c < d 
<< * & a
<<* àáâã
& a
<< à << á << âã
<<< * & p
<<<* PpP
& p
<<< P <<< <<<
<<<< * & k
<<<<* qQ
& k
<<<< q <<<< Q
=* & v
=* VwW
& v
= V = w = W

3.6 Contractions

A multi-character relation string defines a contraction.

Specifying Contractions
Example Description
& k
< ch
Make the sequence 'ch' sort after 'k', as a primary (base-character) difference

3.7 Expansions

<!ELEMENT x (context?, ( p | pc | s | sc | t | tc | i | ic )*, extend? ) >

A mapping to multiple collation elements defines an expansion. This is normally the result of a reset position (and/or preceding relation) that yields multiple collation elements, for example &ae<x or &æ<y .

A relation string can also be followed by / and an extension string. The extension string is mapped to collation elements according to the current state, and the relation string is mapped to the concatenation of the regular CEs and the extension CEs. The extension CEs are not modified, not even their case bits. The extension CEs are not retained for following relations.

For example, &a<z/e maps ‘z’ to an expansion similar to &ae<x . However, the first CE of ‘z’ is primary-after that of ‘a’, and the second CE is exactly that of ‘e’, which yields the order ae < x < af < ag < ... < az < z < b.

The choice of reset-to-expansion vs. use of an extension string can be exploited to affect contextual mappings. For example, &L·=x yields a second CE for ‘x’ equal to the context-sensitive middle-dot-after-L (which is a secondary CE in the root collation). On the other hand, &L=x/· yields a second CE of the middle dot by itself (which is a primary CE).

The two ways of specifying expansions also differ in how case bits are computed. When some of the CEs are copied verbatim from an extension string, then the relation string’s case bits are distributed over a smaller number of normal CEs. For example, &aE=Ch yields an uppercase CE and a lowercase CE, but &a=Ch/E yields a mixed-case CE (for ‘C’ and ‘h’ together) followed by an uppercase CE (copied from ‘E’).

In summary, there are two ways of specifying expansions which produce subtly different mappings. The use of extension strings is unusual but sometimes necessary.

3.8 Context Before

A relation string can have a prefix (context before) which makes the mapping from the relation string to its tailored position conditional on the string occurring after that prefix. For details see the specification of Context-Sensitive Mappings.

For example, suppose that "-" is sorted like the previous vowel. Then one could have rules that take "a-", "e-", and so on. However, that means that every time a very common character (a, e, ...) is encountered, a system will slow down as it looks for possible contractions. An alternative is to indicate that when "-" is encountered, and it comes after an 'a', it sorts like an 'a', and so on.

Specifying Previous Context
Rules
& a <<< a | '-'
& e <<< e | '-'
...

Both the prefix and extension strings can occur in a relation. For example, the following are allowed:

3.9 Placing Characters Before Others

There are certain circumstances where characters need to be placed before a given character, rather than after. This is the case with Pinyin, for example, where certain accented letters are positioned before the base letter. That is accomplished with the following syntax.

&[before 2] a << à

The before-strength can be 1 (primary), 2 (secondary), or 3 (tertiary).

It is an error if the strength of the reset-before differs from the strength of the immediately following relation. Thus the following are errors.

3.10 Logical Reset Positions

The CLDR table (based on UCA) has the following overall structure for weights, going from low to high.

Specifying Logical Positions
Name Description UCA Examples
first tertiary ignorable
...
last tertiary ignorable
p, s, t = ignore Control Codes
Format Characters
Hebrew Points
Tibetan Signs
...
first secondary ignorable
...
last secondary ignorable
p, s = ignore None in UCA
first primary ignorable
...
last primary ignorable
p = ignore Most combining marks
first variable
...
last variable
if alternate = non-ignorable
p != ignore,
if alternate = shifted
p, s, t = ignore
Whitespace,
Punctuation
first regular
...
last regular
p != ignore General Symbols
Currency Symbols
Numbers
Latin
Greek
...
implicits p != ignore, assigned automatically CJK, CJK compatibility (those that are not decomposed)
CJK Extension A, B
Unassigned
first trailing
...
last trailing
p != ignore,
used for trailing syllable components
Jamo Trailing
Jamo Leading

Each of the above Names (except implicits) can be used with a reset to position characters relative to that logical position. That allows characters to be ordered before or after a logical position rather than a specific character.

Note: The reason for this is so that tailorings can be more stable. A future version of the UCA might add characters at any point in the above list. Suppose that you set character X to be after Y. It could be that you want X to come after Y, no matter what future characters are added; or it could be that you just want Y to come after a given logical position, for example, after the last primary ignorable.

Each of these special reset positions always maps to a single collation element.

Here is an example of the syntax:

& [first tertiary ignorable] << à 

For example, to make a character be a secondary ignorable, one can make it be immediately after (at a secondary level) a specific character (like a combining diaeresis), or one can make it be immediately after the last secondary ignorable.

Each special reset position adjusts to the effects of preceding rules, just like normal reset position strings. For example, if a tailoring rule creates a new collation element after &[last variable] (via explicit tailoring after that, or via tailoring after the relevant character), then this new CE becomes the new last variable CE, and is used in following resets to [last variable] .

[last regular] is not actually the last normal CE with a primary weight before implicit primaries. It is used to tailor large numbers of characters, usually CJK, into the script=Hani range between the last regular script and the first implicit CE. (The first group of implicit CEs is for Han characters.) Therefore, [last regular] is set to the first Hani CE, the artificial script boundary CE at the beginning of this range. For example: &[last regular]<*亜唖娃阿...

The [last variable] indicates the "highest" character that is treated as punctuation with alternate handling. Unlike the other logical positions, it can be reset as well as referenced. For example, it can be reset to be just above spaces if all visible punctuation are to be treated as having distinct primary values.

Specifying the Position of [last variable]
Attribute Options Example
variableTop at & x
= [variable top]
after & x
< [variable top]
before & [before 1] x
< [variable top]

The default value for [last variable] is the highest punctuation mark, thus below symbols. The value can be further changed by using the variable-top setting. This takes effect, however, after the rules have been built, and does not affect any characters that are reset relative to the [last variable] value when the rules are being built. The variable-top setting might also be changed via a runtime parameter. That also does not effect the rules.

3.11 Special-Purpose Commands

<!ELEMENT import EMPTY >
<!ATTLIST import source CDATA #REQUIRED >
<!ATTLIST import type CDATA #IMPLIED >

The import command imports rules from another collation. This allows for better maintenance and smaller rule sizes. The source is the locale of the source, and the type is the type (if any). If the source is "locale" it is the same locale. The type is defaulted to "standard".

Example:

<import source="de" type="phonebook"/>
Special-Purpose Commands
Rule Syntax Sub-Element of collation (preferred)
[suppressContractions [Љ-ґ]] <suppress_contractions>[Љ-ґ] </suppress_contractions>
[optimize [Ά-ώ]] <optimize>[Ά-ώ] </optimize>

The suppress contractions tailoring command turns off any existing contractions that begin with those characters, as well as any prefixes for those characters. It is typically used to turn off the Cyrillic contractions in the UCA, since they are not used in many languages and have a considerable performance penalty. The argument is a Unicode Set.

The suppress contractions command has immediate effect on the current set of mappings, including mappings added by preceding rules. Following rules are processed after removing any context-sensitive mappings originating from any of the characters in the set.

The optimize tailoring command is purely for performance. It indicates that those characters are sufficiently common in the target language for the tailoring that their performance should be enhanced.

The reason that these are not settings is so that their contents can be arbitrary characters.


Example:

The following is a simple example that combines portions of different tailorings for illustration. For more complete examples, see the actual locale data: Japanese, Chinese, Swedish, and German (type="phonebook") are particularly illustrative.

<collation>
  <settings caseLevel="on"/>
  <cr><![CDATA[
    &Z
    < æ <<< Æ
    < å <<< Å <<< aa <<< aA <<< Aa <<< AA
    < ä <<< Ä
    < ö <<< Ö << ű <<< Ű
    < ő <<< Ő << ø <<< Ø
    &V <<<* wW
    &Y <<<* üÜ
    &[last non-ignorable]
    # The following is equivalent to <亜<唖<娃...
    <* 亜唖娃阿哀愛挨姶逢葵茜穐悪握渥旭葦芦
    <* 鯵梓圧斡扱
  ]]></cr>
</collation>

3.12 Collation Reordering

Collation reordering allows scripts and certain other defined blocks of characters to be moved relative to each other parametrically, without changing the detailed rules for all the characters involved. This reordering is done on top of any specific ordering rules within the script or block currently in effect. Reordering can specify groups to be placed at the start and/or the end of the collation order. For example, to reorder Greek characters before Latin characters, and digits afterwards (but before other scripts), the following can be used:

Rule Syntax Sub-Element of collation (preferred) Locale Identifier
[reorder Grek Latn digit] <settings reorder="Grek Latn digit"/> en-u-kr-grek-latn-digit

In each case, a sequence of reorder_codes is used, separated by spaces in the settings attribute and in rule syntax, and by hyphens in locale identifiers.

A reorder_code is any of the following special codes:

  1. space, punct, symbol, currency, digit - core groups of characters below 'a'
  2. any script code from the Recommended Table in UAX 31 except Katakana, Common, and Inherited.
    1. Katakana characters are are always reordered with Hiragana.
    2. Characters in any script not in the Recommended Table are treated as being in the preceding Recommended script, in DUCET order. Thus Phoenician characters always reorder with Hebrew characters.
  3. others - where all codes not explicitly mentioned should be ordered. The script code Zzzz (Unknown Script) is a synonym for others.

It is an error if a code occurs multiple times.

Interpretation of a reordering list

The reordering list is interpreted as if it were processed in the following way.

  1. If any core code is not present, then it is inserted at the front of the list in the order given above.
  2. If the others code is not present, then it is inserted at the end of the list.
  3. The others code is replaced by the list of all script codes not explicitly mentioned, in DUCET order.
  4. The reordering list is now complete, and used to reorder characters in collation accordingly.

The locale data may have a particular ordering. For example, the Czech locale data could put digits after all letters, with [reorder others digit] . Any reordering codes specified on top of that (such as with a bcp47 locale identifier) completely replace what was there. To specify a version of collation that completely resets any existing reordering to the DUCET ordering, the single code others can be used, as below.

Examples:

Locale Identifier Effect
en-u-kr-latn-digit Reorder digits after Latin characters (but before other scripts like Cyrillic).
en-u-kr-others-digit Reorder digits after all other characters.
en-u-kr-arab-cyrl-others-symbol Reorder Arabic characters first, then Cyrillic, and put symbols at the end—after all other characters.
en-u-kr-others Remove any locale-specific reordering, and use DUCET order for reordering blocks.

The default reordering groups are defined by the FractionalUCA.txt file, based on the primary weights of associated collation elements. The [top_byte] table contains a mapping from the first (top) byte of primary weights to the associated reordering group. For example:

U+02D0 MODIFIER LETTER TRIANGULAR COLON has a fractional UCA collation weight of [0E 0B, 05, 05]. In the [top_byte] table, the line [top_byte 0E SYMBOL] indicates that 0E maps to SYMBOL.

There are some special cases:

The default reordering groups follow the allkeys_CLDR.txt ordering; they also may be tailored by implementations to different values. For more information on FractionalUCA.txt and allkeys_CLDR.txt, see Collation Auxiliary.

The DUCET ordering is slightly different from the allkeys_CLDR ordering. The reordering groups for the DUCET are not specified here. However, most reordering groups would start with the same characters as in FractionalUCA.txt.

3.13 Case Parameters

The case level is an optional intermediate level ("2.5") between Level 2 and Level 3 (or after Level 1, if there is no Level 2 due to strength settings). The case level is used to support two parametric features: ignoring non-case variants (Level 3 differences) except for case, and giving case differences a higher-level priority than other tertiary differences. Distinctions between small and large Kana characters are also included as case differences, to support Japanese collation.

The case first parameter controls whether to swap the order of upper and lowercase. It can be used with or without the case level.

Importantly, the case parameters have no effect in many instances. For example, they have no effect on the comparison of two non-ignorable characters with different primary weights, or with different secondary weights if the strength = secondary (or higher).

When either the case level or case first parameters are set, the following describes the derivation of the modified collation elements. It assumes the original levels for the code point are [p.s.t] (primary, secondary, tertiary). This derivation may change in future versions of LDML, to track the case characteristics more closely.

Untailored Characters

For untailored characters and strings, that is, for mappings in the root collation, the case value for each collation element is computed from the tertiary weight listed in allkeys_CLDR.txt. This is used to modify the collation element.

  1. If the character is U+FFFE (lowest-weight), set case value = LOWEST.
  2. Otherwise, look up a case value for the tertiary weight x of each collation element:
    1. UPPER if x ∈ {08-0C, 0E, 11, 12, 1D}
    2. UNCASED otherwise
    3. FractionalUCA.txt encodes the case information in bits 6 and 7 of the first byte in each tertiary weight. The case bits are set to 00 for UNCASED, LOWEST and LOWERCASE, and 10 for UPPER. There is no MIXED case value (01) in the root collation.

Compute Modified Collation Elements

From a computed case value, set a weight c according to the following.

  1. If the value is LOWEST (U+FEFF), set c = 1
  2. Otherwise if CaseFirst=UpperFirst, set c = UPPER ? 2 : MIXED ? 3 : 4
  3. Otherwise set c = UPPER ? 4 : MIXED ? 3 : 2

Compute a new collation element according to the following table. The notation xt means that the values are numerically combined into a single level, such that xt < yu whenever x < y. The fourth level (if it exists) is unaffected.

Case Level Strength Original CE Modified CE Comment
on primary 0.s.t 0.0 ignore case level weights of primary-ignorable CEs
p.s.t p.c
secondary
or higher
0.0.t 0.0.0.t ignore case level weights of secondary-ignorable CEs
0.s.t 0.s.c.t
p.s.t p.s.c.t
off any 0.0.0 0.0.00 ignore case level weights of tertiary-ignorable CEs
0.0.t 0.0.4t
0.s.t 0.s.ct
p.s.t p.s.ct

Note the special case weights when s = 0. They ensure the construction of well-formed case and tertiay weights. For details, see Section 3.7, Well-Formed Collation Element Tables in [UCA].

Tailored Strings

Characters and strings that are tailored have case values computed from their root collation case bits.

  1. Look up the tailored string’s root CEs. (Ignore any prefix or extension strings.) N=number of primary root CEs.
  2. Determine the number and type (primary vs. weaker) of CEs a tailored string maps to. M=number of primary tailored CEs.
  3. If N<=M (no more root than tailoring primary CEs): Copy the root case bits for primary CEs 0..N-1.
    • If N<M (fewer root primary CEs): Clear the case bits of the remaining tailored primary CEs. (uncased/lowercase/small Kana)
  4. If N>M (more root primary CEs): Copy the root case bits for primary CEs 0..M-2. Set the case bits for tailored primary CE M-1 according to the remaining root primary CEs M-1..N-1:
    • Set to uncased/lower if all remaining root primary CEs have uncased/lower.
    • Set to uppercase if all remaining root primary CEs have uppercase.
    • Otherwise, set to mixed.
  5. Clear the case bits for secondary CEs 0.s.t.
  6. Tertiary CEs 0.0.t must get uppercase bits.
  7. Tertiary-ignorable CEs 0.0.0 must get ignorable-case=lowercase bits.

Note: Almost all Cased characters have primary (non-ignorable) root collation CEs, except for U+0345 Combining Ypogegrammeni which is Lowercase. All Uppercase characters have primary root collation CEs.

3.14 Visibility

<!ATTLIST collation visibility ( internal | external ) "external" >

Collators have external visibility by default, meaning that they can be displayed in a list of collation options for users to choose from. Collators marked as having internal visibility should not be shown in such a list. Collators are typically internal when they are partial sequences included in other collators.

3.15 Collation Indexes

Index Characters

The main data includes <exemplarCharacters> for collation indexes. See the main document, Section 5.6 Character Elements, for general information about exemplar characters.

The index characters are a set of characters for use as a UI "index", that is, a list of clickable characters (or character sequences) that allow the user to see a segment of a larger "target" list. Each character corresponds to a bucket in the target list. One may have different kinds of index lists; one that produces an index list that is relatively static, and the other is a list that produces roughly equally-sized buckets. While CLDR is mostly focused on the first, there is provision for supporting the second as well.

The index characters need to be used in conjunction with a collation for the locale, which will determine the order of the characters. It will also determine which index characters show up.

The static list would be presented as something like the following (either vertically or horizontally):

… A B C D E F G H CH I J K L M N O P Q R S T U V W X Y Z …

In the "A" bucket, you would find all items that are primary greater than or equal to "A" in collation order, and primary less than "B". The use of the list requires that the target list be sorted according to the locale that is used to create that list. Although we say "character" above, the index character could be a sequence, like "CH" above. The index exemplar characters must always be used with a collation appropriate for the locale. Any characters that do not have primary differences from others in the set should be removed.

Details:

  1. The primary weight (according to the collation) is used to determine which bucket a string is in. There are special buckets for before the first character, between buckets of different scripts, and after the last bucket (and of a different script).
  2. Characters in the index characters do not need to have distinct primary weights. That is, the index characters are adapted to the underlying collation: normally Ё is in the Е bucket for Russian, but if someone used a variant of Russian collation that distinguished them on a primary level, then Ё would show up as its own bucket.
  3. If an index character string ends with a single "*" (U+002A), for example "Sch*" and "St*" in German, then there will be a separate bucket for the string minus the "*", for example "Sch" and "St", even if that string does not sort distinctly.
  4. An index character can have multiple primary weights, for example "Æ" and "Sch". Names that have the same initial primary weights sort into this index character’s bucket. This can be achieved by using an upper-boundary string that is the concatenation of the index character and U+FFFF, for example "Æ\uFFFF" and "Sch\uFFFF". Names that sort greater than this upper boundary but less than the next index character are redirected to the last preceding single-primary index character (A and S for the examples here).

For example, for index characters [A Æ B R S {Sch*} {St*} T] the following sample names are sorted into an index as shown.

The … items are special: each is a bucket for everything else, either less or greater. They are inserted at the start and end of the index list, and on script boundaries. These are really script boundaries, not reordering code boundaries. Each script has its own range, except where scripts sort primary-equal (e.g., Hira & Kana). All characters that sort in the low reordering groups (whitespace, punctuation, symbols, currency symbols, digits) are treated as a single script for this purpose. So if you had a collation that reordered Hebrew after Ethiopic, you would still get index boundaries between the following (and in that order):

  1. Ethiopic
  2. Hebrew
  3. Phoenician // included in the Hebrew reordering group
  4. Samaritan // included in the Hebrew reordering group
  5. Devanagari

If you tailor a Greek character into the Cyrillic script, that Greek character will be bucketed (and sorted) among the Cyrillic ones.

In the UI, an index character could also be omitted or grayed out if its bucket is empty. For example, if there is nothing in the bucket for Q, then Q could be omitted. That would be up to the implementation. Additional buckets could be added if other characters are present. For example, we might see something like the following:

Sample Greek Index
Contents
 Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω
With only content beginning with Greek letters 
 … Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω …
With some content before or after
 … 9 Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω …
With numbers, and nothing between 9 and Alpha
  … 9 A-Z Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω …
With numbers, some Latin

Here is a sample of the XML structure:

<exemplarCharacters type="index">[A B C D E F G H I J K L M N O P Q R S T U V W X Y Z]</exemplarCharacters>

The display of the index characters can be modified with the Index labels elements, discussed in the main document, Section 5.6.4 Index Labels .

CJK Index Markers

Special index markers have been added to the CJK collations for stroke, pinyin, zhuyin, and unihan. These markers allow for effective and robust use of indexes for these collations. For example, near the start of the pinyin tailoring there is the following:

<p> A</p><!-- INDEX A -->
<pc>阿呵𥥩锕𠼞𨉚</pc><!-- ā -->

<pc>翶</pc><!-- ao -->
<p> B</p><!-- INDEX B -->

These indicate the boundaries of "buckets" that can be used for indexing. They are always two characters starting with the noncharacter U+FDD0, and thus will not occur in normal text. For pinyin the second character is A-Z; for unihan it is one of the radicals; and for stroke it is a character after U+2800 indicating the number of strokes, such as ⠁. For zhuyin the second character is one of the standard Bopomofo characters in the range U+3105 through U+3129.

The corresponding bucket label strings are the boundary strings with the leading U+FDD0 removed. For example, the Pinyin boundary string "\uFDD0A" yields the label string "A".

However, for stroke order, the label string is the stroke count (second character minus U+2800) as a decimal-digit number followed by 劃 (U+5283). For example, the stroke order boundary string "\uFDD0\u2805" yields the label string "5劃".