|Authors||Asmus Freytag (firstname.lastname@example.org)|
This report identifies a set of operations that map similar characters to a common target. Such operations, called character foldings are used to ignore certain distinctions between similar characters. The report also provides an algorithm for applying these operations to searching and additional guidelines.
This document is a Draft Unicode Technical Report. Publication does not imply endorsement by the Unicode Consortium. This is a draft document which may be updated, replaced, or superseded by other documents at any time. This is not a stable document; it is inappropriate to cite this document as other than a work in progress.
A Unicode Technical Report (UTR) contains informative material. Conformance to the Unicode Standard does not imply conformance to any UTR. Other specifications, however, are free to make normative references to a UTR.
Please submit corrigenda and other comments with the online reporting form [Feedback]. Related information that is useful in understanding this document is found in [References]. For the latest version of the Unicode Standard see [Unicode]. For a list of current Unicode Technical Reports see [Reports]. For more information about versions of the Unicode Standard, see [Versions].
The foldings specified in this Technical Report cover Unicode, Version 4.0
This report identifies a set of character foldings, in other words, operations that map similar characters to a common target. Folding operations are most often used to temporarily ignore certain distinctions between similar characters. For example, they are are useful for "fuzzy" or "loose" searches. More rarely, certain folding operations may be used to permanently remove distinctions. Each of the folding operations specified in this report has well-understood properties, and is appropriate in specific contexts. For example some identifiers need case folding, some do not. Some text searches need to preserve superscript forms such as the trademark symbol ™, while others do not. For those and similar reasons, not all of these folding operations may be appropriate in a given context. See Section 5.2 for some of the more problematic folding or expansion operations.
The report also a provides an algorithm for combining these operations for the purpose of searching or programmatic identifier matching. This algorithm combines canonical normalization with optional folding operations. This allows implementers to decide which folding option is useful for a particular purpose.
Repeatedly applying the same folding does not change the result, a property called idempotency. Since foldings remove distinctions, they lose information. For that reason it is not possible to construct an inverse operation, except in the trivial case of an identity folding.
Foldings have a domain of operation. All characters not in this domain are left unchanged. Two foldings that otherwise perform the same operation are distinct if their domain of operation is different.
Foldings can be applied transiently, for example the same folding can be applied to two strings before comparison, or they can be used to permanently transform a text, for example when applying the Positional Forms Folding to converting legacy data that uses explicit Arabic positional shapes to the generic Arabic characters with implicit directionality.
In a more general sense, the elements of equivalence classes or the target of a folding may be character sequences, such as combining character sequences. An example are the Katakana foldings where voiced syllables are written with two characters for halfwidth Katakana and single characters otherwise (ｶ + ﾞ → ガ). A character may be folded differently when it is part of different character sequences or when it is by itself. This means that foldings in general are context sensitive. Finally, the output of a folding operation, whether context sensitive or not, may result in a string that is longer or shorter than the input string.
For the purpose of fuzzy text matching, including both programmatic identifier matching and general text searching, it is often necessary to selectively ignore otherwise meaningful distinctions between related characters, for example upper and lower case, presence of accent marks, etc. This process can be called search term folding. Depending on the search operation, different foldings need to be applied, and possible interactions between foldings and between folded characters and adjacent text that is not folded must be carefully managed, as they can affect the result of the search and introduce both false positive or false negative matches. The remainder of the document describes various foldings and discusses their use in the context of search term folding.
A significant aspect of string foldings for programmatic identifier matching is that the set of allowable identifier characters is restricted. Issues that relate to identifier matching in particular are addressed specifically below.
Normalization [Normalization] is part of any robust search term folding algorithm (see section Section 4.1.1 Basic Folding Algorithm). However, there are some important differences between normalization and the foldings that make up search term folding. Normalization, particular the canonical forms (NFC or NFD) are often intended for permanent transformation of data, while search term and other foldings are by nature transient. Further, unlike most of the other foldings considered here, normalization it is not context-independent, since the equivalences are not between characters, but character sequences.
As defined, the normalization forms offer only two broad levels of distinctions (they either preserve or do not preserve compatibility distinctions). The choice of which distinctions may be ignored for search term folding needs to be more specific; it depends on the nature of the operation. One size does not fit all.
For example, two of the normalization forms depend on compatibility mappings which replace character by their compatibility decompositions. Applying certain of these compatibility mappings may lead to unintended false positive matches, preventing their use in general text searches. In combination with whole word search it could even lead to unintended false negatives. (See Section 5.2 Problematical foldings).
Furthermore, normalization and case folding are defined as separate and independent operations, but case folding often occurs together with other foldings in search term folding. In order to avoid inconsistencies, search term folding needs to address the interaction of case folding with the other steps in the algorithm.
Search term folding includes canonical normalization; however, whether the normalized text is in the composed (NFC) or decomposed form (NFD) is of secondary importance in terms of defining the foldings. Due to the transient nature of search term folding, the distinction between NFC and NFD is immaterial, as long as the two forms are not mixed. However, if the data is known to be in one of the normalized forms, it would be computationally less expensive to operate the search in that form.
Like foldings, the comparisons at the heart of the Unicode Collation Algorithm [UCA] also define equivalences. One can derive a specific folding by applying the collation algorithm with a particular strength and specific tailorings. Foldings derived in this manner can be useful in searches that ignore similar distinctions to those ignored in collation. Such foldings are not subject of this report.
This technical report contains no formal definitions. For formal definitions of character properties and related terms, including string function and folding function see UTR #23, Unicode Character Property Model [PropModel]. All other terms are used as defined in the [Unicode], particularly in chapter 3, Conformance, or in the online [Glossary].
The following notational conventions are used in this TR:
|XXXX..YYYY||indicates an inclusive range|
|XXXX, YYYY||indicates an alternative|
|<cccc>||refers to a compatibility mapping tag as defined in [Aliases]. This should not be confused with a character code sequence of length 1, which would be <XXXX> where X is an upper case hex digit.|
|Xy||refers to a particular value for the General Category property defined in [UnicodeData] , e.g. "Pd"|
|<+>||means a folding is contained in the Unicode data files, but its general use is not recommended|
All specifications of algorithms are in terms of results—all implementations that achieve the same result are fully equivalent. In particular, implementations commonly use optimization techniques, such as normalizing and folding 'on demand'. In each of the following algorithms, implementations may be able to avoid the loop implied by step (c) by perfoming additional transformations whose effect ensures the the folding is stable under normalization.
The basic algorithm for search term folding can be stated as
a. Apply optional folding operations
b. Apply canonical decomposition
c. Repeat (a) and (b) until stable
d. Apply composition if necessary
where each step is applied on the whole string, and applies to the result of the preceding operation.
For identifier folding, it is important to account for prohibited characters. This adds a new step (e). The modified basic algorithm then becomes:
a. Apply optional folding operations
b. Apply canonical decomposition
c. Repeat (a) and (b) until stable
d. Apply composition if necessary
e. Eliminate or flag forbidden characters
Foldings in step (a) can be modified to also disallow certain characters by mapping them to forbidden characters, which are then caught in step (e).
The following table summarizes the definition of a number of important and well-defined folding operations for which the data are available in the Unicode Character Database [UCD] or as datafiles associated with this Technical Report. Foldings that are multigraph expansions have been collected together. Such a folding replaces a digraph or higher multigraph by its expansion into an equivalent sequence of base characters, such as replacing DOUBLE PRIME or TRIPLE PRIME by two or three PRIME characters respectively. The foldings listed at the end of the table are provisional: only a provisional definition exists, and in some cases no associated datafile.
|Descriptive Name||Source Characters||Target Characters||Data file specifying the mapping|
|Accent removal||Latin/Greek/Cyrillic characters with canonical decomposition||base characters of [CD]||[UnicodeData]|
|Case folding||any||case fold according to CaseFolding.txt||[CaseFolding]|
|Canonical duplicates folding (e.g. Ohm - Omega)||0374, 037E, 0387, 1FBE, 1FEF, 1FFD, 2000, 2001, 2126, 212A, 212B, 2329..232A||[CD]||[UnicodeData]|
|Greek letterforms folding||03D0..03D2, 03D5..03D6, 03F0..03F2, 03F4..03F5||[KD]||[UnicodeData]|
|Hebrew Alternates folding||FB20..FB28||[KD]||[UnicodeData]|
|Math symbol folding||<font>||[KD]||[UnicodeData]|
|Native digit folding||Nd||substitute ASCII digit of same numeric property||[UnicodeData]|
|Overline folding||FE49..FE4C||[KD] maps to:
|Positional Forms folding
- includes Arabic ligatures
|<initial>, <medial>, <final>, <isolate>||[KD]||[UnicodeData]|
|Small forms folding||<small>||[KD]||[UnicodeData]|
|Spacing Accents <+>||00AF,00B4,00B8,02D8..02DD, 037A,0384,1FBD,1FBE..1FC0, 1FFE,2017,203E,309B..309C||[KD]||[UnicodeData]|
|Symbol folding <+>||00B5, 2107,2135..2138||[KD]||[UnicodeData]|
|Underline folding||2017, FE4D..FE4F||005E||[UnicodeData]|
|Vertical forms folding||<vertical>||[KD]||[UnicodeData]|
|- Circled symbols expansion||<circled>||[KD]||[UnicodeData]|
|- Ellipsis expansion||2024..2026||[KD]||[UnicodeData]|
|- Fraction expansion||<fraction>||[KD]||[UnicodeData]|
|- Integral expansion||222C..222D,222F..2230||[KD]||[UnicodeData]|
|- Ligature expansion Misc.||0587, 0675..0678, 0E33, 0EB3, 0EDC..0EDD, 0F77, 0F79, FB00..FB06, FB13..FB17, FB4F||[KD]||[UnicodeData]|
|- Parenthesized||2474..2487,249C..24B5, 3200..3243||[KD]||[UnicodeData]|
|- Prime expansion||2033..2034,2036..2037||[KD]||[UnicodeData]|
|- Roman numerals||2160..2183||[KD]||[UnicodeData]|
|- Squared (unmarked)||3358..3370, 33E0..33FE, 32C0..32CB||[KD]||[UnicodeData]|
|- Digraphs||0132..0133, 013F..0140. 0149, 01C4..01CC, 01F1..01F3, 1E9A||[KD]||[UnicodeData]|
| - Other multigraphs,
e.g. c/o, TEL
|203C, 2047..2049, 20A8, 2100..2101, 2103, 2105..2106, 2109, 2116, 2121||[KD]||[UnicodeData]|
|Diacritic removal (includes stroke, hook, descender)||any||fold Latin/Greek/Cyrillic characters with diacritics to their related base character according to DiacriticFolding.txt||[DiacriticFolding] TBD|
|Han Radical folding||2F00..2F5D, 2EF3, 2E9F||TBD||[RadicalFolding] TBD|
|Letterforms folding||Variants of letter forms
e.g. 017F (long s)
|fold to related base form
e.g. 0073 (s)
|Simplified Han Folding||simplified Han characters||TBD (corresponding traditional characters)||[SimplifiedFolding] TBD|
|Superscript folding||<super>, plus modifier letters 02C0..02C1,06E5..06E6,1D2C..1D61
plus other modifier letters
|[KD] with some additions||[SuperScriptFolding]|
|Suzhou Numeral folding||3038..303A, 3021..3029||TBD||[SuzhouFolding] TBD|
|Width folding||<wide>,<narrow>any||[KD] with additional handling of contraction for narrow kana sound marks||[WidthFolding]|
The most important guideline is "discriminate". Understand the effect before applying a folding. Do not apply any of these folding all at once just because it exists, and certainly never all of them at once. The following notes on individual foldings or issues may be of help in following this guideline.
Diacritic folding goes beyond the decomposition and removal of accents, umlauts, cedillas etc. that is provided by the canonical decompositions, but also includes barred, slashed forms etc, as well as hooks, descenders, etc.
Greek letter forms should be folded for Greek text. They should not be folded for mathematical and scientific usage as doing so would conflate very distinct concepts. To give an example of common usage consider physics, which distinguishes angle encoded by theta "θ" and temperature encoded by theta symbol "ϑ".
The compatibility decompositions contain only a subset of variant letter forms that could be folded for search purposes, including Greek final sigma and Latin long s.
Greek final sigma should not be folded, unless transiently. Latin long s should be folded for modern text in roman type style but, other than transiently for searches, should not be folded for texts intended to be set in Fraktur type. Other letterform foldings include alternate Hebrew characters.
There are a number of foldings applicable to Han Ideographs. While the Unicode Consortium has not yet published any datafile defining these, they can be described in general terms.
Han Radical folding substitutes a Unified ideograph for a Han radical.
Simplified and traditional forms of Han ideographs are separately encoded in Unicode, even where they represent the same meaning. This folding would remove the distinction. In practice, there are additional differences in Han character usage between writers of simplified and traditional Han ideographs, some of which cannot be folded without context and semantic information. A simple 1:1folding is nevertheless useful for many purposes.
As results of historical development of the Han ideograph there are multiple variations of characters for the same concept, for example there are 47 variants for 'turtle'. Such folding would remove such variations. The process of defining such a folding for all such cases will be difficult and lengthy.
The Unicode Standard maintains duplications for certain Han ideographs based on the fact that they were separately encoded within a given source character set. A Han source separation folding would equivalence such separated characters.
Japanese Katakana and Hiragana are two generally equivalent syllabaries, where each Hiragana syllable has a corresponding Katakana syllable. Foldings in both directions are useful, depending on the situation. However, since Katakana are used to represent the pronunciation of foreign words in Japanese, there are more Katakana than Hiragana characters.
There is one important difference in orthography between the syllabaries, affecting the way the long syllables are expressed. Hiragana uses an additional vowel, while Katakana uses a length mark. If this is taken into account, the folding is no longer context free. In fact, these are better seen as examples of algorithmic transliterations, such as the ISCII transliterations between Indic scripts.
In addition, Katakana occur in both regular and halfwidth forms, with the halfwidth forms using two characters to express voiced or semi-voiced syllables, where the regular Katakana use a single character. The [HiraganaFolding] merely folds Hiragana to regular Katakana, while the [KatakanaFolding] folds both halfwidth and regular Katakana characters to Hiragana.
Ethiopic is an example of an "open alphasyllabary" where a single symbol represents a CV (consonant + vowel) pattern. In principle, such a syllabary has several forms for each consonant C: one for each of the vowels V, and one vowel-less form. However, not all forms are actually used in a given syllabary.
Vowels often appear in the wrong form in electronic text due to typographical errors (e.g. from incompatible input methods), different spelling conventions, or where grammatical word inflections are primarily expressed by a change of vowel, as it the case for Ethiopic. In such situations it may make sense to fold away the vowel by converting all syllables in a string that have the same consonant into a common reference form.
The choice of reference form depends on the syllabary, or more precisely on features of the languages that are using the syllabary. For Ethiopic, a vowel-less form has been found to be the most practical target for folding.
For other syllabaries, vowel folding may not be as useful.
Semantically neutral foldings could be defined as those foldings that simply remove a distinction that is more or less purely an artifact of the encoding itself. Under the right circumstances, these foldings are in principle candidates for permanent data transformation. This is primarily true for the canonical decomposition and composition, but could also apply to text using the Arabic positional forms. If such a text is converted to use non-positional forms, but rendered via a standard Unicode rendering process, the appearance would be the same (except for deliberately odd combinations of positional shapes). In practice it is far more likely that the data containing the positional forms will display poorly on a system that expects characters with implicit positional shaping.
The following foldings can be considered semantically neutral
Note that Arabic positional folding, especially when intended as a permanent data transformation, may need to introduce ZWJ or ZWNJ characters.
Foldings based on tailored collation data would fold characters that are 'nearly equivalent' in a particular language. For example, a locale-based folding for Swedish could follow common practice in Sweden and match the following pairs of character sequences, among others, based on equivalence in pronunciation.
In other words, locale-based foldings would be different for some user groups using the same script (in this case Latin). The recommended way to implement locale-based searching based on sorting tables is found in [UCA ].
Compatibility decomposition provides a fixed combination of several foldings and expansions. It is in fact the source of most of the foldings in the table in section 4.0. There are two ways to subdivide compatibility decompositions:
The specifications in Section 5.0 uses the first method, whenever the compatibility tag is well defined and meaningful. Where it is too broad, e.g., for the <compat> tag, foldings are further subdivided by defining specific ranges of source characters.
Using compatibility decomposition is convenient, since existing algorithms for Normalization may provide them. However, the full decomposition includes several foldings that may not be appropriate for the given purpose. By selectively not applying the decomposition to certain character ranges given in Section 4.0, one can in effect limit the compatibility decomposition to only the desired foldings.
Some foldings can have unintended consequences, including inadvertent changes in the semantics of the text. In most cases, it is best to be conservative and avoid problematic foldings altogether. There are two general exceptions to this rule. The first is the case of identifier matching. If a folding has a prohibited character as one of the output characters, it will not match any legal identifier. Therefore, for properly restricted inputs, one may safely use fixed combinations of foldings, such as NFKC. The other exception is the case of more extensive string pre-processing, discussed below.
Fraction expansion as defined in the compatibility decompositions can lead to a drastic change of the semantics of a string and can lead to term boundary issues for searching. For example: Expanding the fraction in this string: <DIGIT 5, VULGAR FRACTION ONE QUARTER> turns it into <DIGIT 5, DIGIT 1, FRACTION SLASH, DIGIT 4>. This now will be found by a search for "51". Because of the semantics of FRACTION SLASH the expansion also changes the numeric value from "5 and a quarter" into "51 over 4". Fraction expansion is therefore best avoided altogether.
By modifying the fraction expansion from the standard compatibility decomposition and inserting an appropriate space character, for example THIN SPACE, before the fraction, it is possible to prevent the expanded fraction from coalescing with preceding digits. However, if there are no preceding digits, no THIN SPACE must be added, or strings containing the expanded fraction would no longer match strings with already expanded fractions which presumably would not contain THIN SPACE characters. Finally, any space character would be subject to space folding, which, if present would introduce a SPACE character, possibly affecting the search term.
If a circled bullet character is simply replaced by its contents, as when CIRCLED DIGIT 5 is replaced by DIGIT 5, the separation from the surrounding text is lost, and the DIGIT 5 could run together with adjacent numbers. For bullet characters using parenthesized or dotted letters or digit, this issue is somewhat mitigated by fact that the bullet itself contains punctuation. Bullet characters are commonly used like footnote marks to refer to other text, in other words, they do not occur just at the beginning of bulleted lines.
Spacing accents are mapped by compatibility decomposition to SPACE followed by a non-spacing accent. This inappropriately introduces a space character into the term, as well as introducing non-spacing marks where none were in the data before. The former is especially problematic, where the matching operation is affected by these spaces and combining characters.
The set of compatibility decompositions includes the folding of letterlike mathematical symbols to their nearest ASCII or Hebrew equivalent. In particular the Hebrew characters used as letterlike symbols do not have RIGHT TO LEFT directionality and the set of such letters in mathematical usage is sufficiently restricted that such folding makes little sense in math, except in pure 'looks like' style searches.
Unicode contains many clusters, e.g. square symbols, some of the letterlike characters that are made up of several characters. 'Decomposing' these may or may not be the right thing for search equivalence. Parenthesized characters and numbers would probably be immune to the term boundaries issues raised earlier, but the story is less clear for others.
For more information on Jamo expansion see Section 11.4 Hangul in [Unicode].
To a limited extent, the problems surrounding bullet expansion can be mitigated by inserting a THIN SPACE around the expansion to set the expanded text off from the surrounding text. However, this cannot be applied together with any space folding, as otherwise the THIN SPACE may become SPACE and might be considered a search term delimiter.
As characters are added to [Unicode] many of them need to be added to the definition of foldings described here. However, there are some exceptions, for example, few characters subject to [WidthFolding] are expected to be added. The datafiles that are specifically associated with this Technical Report contain a note discussing the expected level and type of future changes.
No attempt is made to version the datafiles associated with this Technical Report. Each version replaces the preceding version. Each File will indicate which version of the Unicode Standard is required to cover all character codes referred to. In order to recover folding tables for earlier versions of the Standard, simply delete any lines that refer to any characters (whether as source or target characters) for which the [DerivedAge] is more recent than the desired version.
Where significant changes have been made to the folding data for existing characters they are noted in the change history in each data file.
Foldings derived from data in the Unicode Character Database [UCD] can be fully versioned. In the future, datafiles that are originally associated with this Technical Report may be incorporated into the UCD.
|[Aliases]||Data files ftp://ftp.unicode.org/Public/UNIDATA/PropertyAliases.txt and http://www.unicode.org/Public/UNIDATA/PropertyValueAliases.txt|
|[CaseFolding]||Data file ftp://ftp.unicode.org/Public/UNIDATA/CaseFolding.txt|
|[Charts]||The online code charts can be found at http://www.unicode.org/charts/ An index to characters names with links to the corresponding chart is found at http://www.unicode.org/charts/charindex.html|
|[DerivedAge]||The version for which a given character was
added to the Unicode Standard is listed in
|[DiacriticRemoval]||Ed. The data file containing the data for Diacritic removal is TBD.|
|[EAW]||Unicode Standard Annex #11, East
Asian Width. http://www.unicode.org/reports/tr11
For a definition of East Asian Width
|Feedback]||Reporting Errors and Requesting
|[FAQ]||Unicode Frequently Asked Questions
For answers to common questions on technical issues.
For explanations of terminology used in this and other documents.
|[HiraganaFolding]||A data file can be found at: http://www.unicode.org/reports/tr30/datafiles/HiraganaFolding.txt.|
|[Normalization]||Unicode Standard Annex #15: Unicode
|[KatakanaFolding]||A data file can be found at: http://www.unicode.org/reports/tr30/datafiles/KatakanaFolding.txt. Another example of a Hiragana_Katakana transliteration can be found as part of the ICU4j source code.|
|[PropModel]||Unicode Technical Report #23:The Unicode Character Property Model, http://www.unicode.org/reports/tr23/|
|[Reports]||Unicode Technical Reports
For information on the status and development process for technical reports, and for a list of technical reports.
|[SimplifiedFolding]||Ed.: The data file containing the data for simplified to traditional folding is TBD.|
|[SuperScriptFolding]||A data file can be found at: http://www.unicode.org/reports/tr30/datafiles/SuperscriptFolding.txt.|
|[SuzhouFolding]||Ed.: The data file containing the data for Suzhou Folding is TBD.|
|[Unicode]||The Unicode Standard, Version 4.0, (Boston, MA, Addison-Wesley, 2003, ISBN: 0-321-18578-1) or online as http://www.unicode.org/versions/Unicode4.0.0/|
|[UCA]||Unicode Technical Standard #10: Unicode
|[UCD]||Unicode Character Database. http://www.unicode.org/Public/UNIDATA/UnicodeCharacterDatabase.html
For and overview of the Unicode Character Database and a list of its associated files
This file contains the combining class and decomposition information needed to carry out canonical and compatibility decompositions as defined in chapter 3 of the Unicode Standard.
|[UXML]||Unicode Technical Report #20: Unicode
in XML and other Markup Languages
|[Versions]||Versions of the Unicode Standard
For details on the precise contents of each version of the Unicode Standard, and how to cite them.
|[WidthFolding]||A data file can be found at: http://www.unicode.org/reports/tr30/datafiles/WidthFolding.txt. Another example of a Fullwidth_Halfwidth folding can be found as part of the ICU4j source code.|
|[XML]||Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, Eve Maler, Eds., Extensible Markup Language (XML) 1.0 (Second Edition), W3C Recommendation 6-October-2000, http://www.w3.org/TR/REC-xml|
Changes from Tracking Number
2 Added a description of Syllabic folding, replaced definitions by pointer to definitions in [PropModel], improved introduction.
1 Updated to Unicode 4.0, updated Status and References, removed conformance section, added detail throughout
0 First version
Unicode and the Unicode logo are trademarks of Unicode, Inc., and are registered in some jurisdictions.