Internet Draft                                          Paul Hoffman

draft-hoffman-i18n-terms-01.txt                           IMC & VPNC

January 17, 2001

Expires in six months


Terminology Used in Internationalization in the IETF


Status of this memo


This document is an Internet-Draft and is in full conformance with all

provisions of Section 10 of RFC2026.


Internet-Drafts are working documents of the Internet Engineering Task

Force (IETF), its areas, and its working groups. Note that other groups

may also distribute working documents as Internet-Drafts.


Internet-Drafts are draft documents valid for a maximum of six months

and may be updated, replaced, or obsoleted by other documents at any

time. It is inappropriate to use Internet-Drafts as reference material

or to cite them other than as "work in progress."


      The list of current Internet-Drafts can be accessed at



      The list of Internet-Draft Shadow Directories can be accessed at





This document provides a glossary of terms used in the IETF when

discussing internationalization. The purpose is to help frame

discussions of internationalization in the various areas of the IETF and

to help introduce the main concepts to IETF participants.


1. Introduction


As [RFC2277] summarizes: "Internationalization is for humans. This means

that protocols are not subject to internationalization; text strings

are." Many protocols throughout the IETF use text strings that are

entered by, or are visible to, humans. It should be possible to make

these text strings readable to anyone, which means that the text must be

able to be displayed in any human language. This is the challenge of



1.1 About this document


Internationalization is discussed in many working groups of the IETF.

However, few working groups have internationalization experts. When

designing or updating protocols, the question often comes up "should we

internationalize this" (or, more likely, "do we have to internationalize



This document gives an overview of internationalization as it applies to

IETF standards work by covering lightly the many aspects of

internationalization and the vocabulary associated with those topics. It

is not meant to be a complete description of internationalization. The

definitions in this document come from many earlier IETF documents and



As in many fields, there is disagreement in the internationalization

community on definitions for many words. The topic of language brings up

particularly passionate opinions for experts and non-experts alike. This

document attempts to define terms that will be most useful to the IETF



Note that this is a very early draft of the document. Many definitions

here will likely change, and some topics may be added. Discussion of

this document is encouraged. Information on the mailing list for this

document can be found at <http://www.imc.org/ietf-i18n-terms/>.


1.2 Foundations for internationalization




A language is a way that humans interact. The use of language occurs in

many forms, the most common of which are writing, vocal, and visual.

Each language form is independent: some languages have a close

relationship between the written and vocal forms, while others have a

looser relationship. [RFC1766] and [RFC1766bis] discuss languages in

more detail.




A script is a collection of symbols used to represent textual

information in one or more writing systems. A script can have many

attributes, such as the number of written characters and the rules for

combining written characters. Most IETF protocols that deal with

languages deal only with scripts, not spoken or visual forms. [RFC2277]

discusses scripts in more detail.


It is common for internationalization novices to mix up the terms

"language" and "script". This can be a problem in protocols that

differentiate the two, such as mail content protocols. Almost all

internationalized protocols deal with scripts (the written systems),

while fewer deal with languages. Many languages can be expressed using

different scripts.


grapheme, phoneme, alphabet


A grapheme is an abstract, atomic written entity of a script. A phoneme

is an abstract, atomic spoken entity of a spoken language. An alphabet

is a script that maps between graphemes and phonemes.




In the IETF, the verb "internationalize" means to add or improve the

handling of international information in a protocol. Many protocols that

handle text only handle one script, the one that contains the letters

used in English text. Internationalizing such a protocol allows

the protocol to handle more scripts, hopefully all of the ones

useful to anyone in the world.




Internationalized applications can handle a wide variety of languages.

Typical users only understand a small number of languages, so the

program must be tailored to interact with users in just the languages

they know. Localization involves not only changing the language

interaction, but also other relevant changes such as display of numbers,

dates, currency, and so on.


i18n, l10n


These are abbreviations for "internationalization" and "localization".

"18" is the number of characters between the "i" and the "n" in

"internationalization", and "10" is the number of characters between the

"l" and the "n" in "localization".



2. Fundamental Terms


This section covers basic topics that are needed for almost anyone who

is involved with internationalization of IETF protocols. Many terms in

this section are based on [IDN-REQ].




A character is a member of a set of elements used for organization,

control, or representation of data. In written form, a language is

expressed in characters. The same set of characters can often be used

to express many languages.


In ISO 10646 [ISO10646], a character is identified by its name, not by

its shape. A name may suggest a meaning, but the character may be used

for representing other meanings as well. A name may suggest a shape, but

that does not imply that just that is commonly used in print.


coded character


A coded character is a character with its coded representation. A coded

character set (CCS) is a set of unambiguous rules that establish a

character set and the relationship between the characters of the set and

their coded representation. The specified set of characters in a CCS is

often called the "repertoire".


character encoding scheme


A character encoding scheme (CES) is a mapping from one or more coded

character sets to a set of octets. Some CESs are associated with a

single CCS; for example, UTF-8 [RFC2279] applies only to ISO 10646.

Other CESs, such as ISO 2022, are associated with many CCSs.




A charset is a method of mapping a sequence of octets to a sequence of

abstract characters. A charset is, in effect, a combination of one or

more CCS with a CES. Charset names are registered by the IANA according

to procedures documented in [RFC2278]. A particular charset may have

different glyphs depending on the language being used.


transfer encoding syntax


A transfer encoding syntax (TES) (sometimes called a transfer encoding

scheme) is a reversible transform of already-encoded data represented in

one or more character encoding schemes. TESs are useful for encoding

types of character data into an another format, usually for allowing new

types of data to be transmitted over legacy protocols. Examples of TESs

used in the IETF include Base64 and quoted-printable (described in more

detail in Chapter 6).




3. Standards Bodies and Standards


This section describes some of the standards bodies and standards that

appear in discussions of internationalization in the IETF. This is an

incomplete and possibly over-full list; listing too few bodies or

standards can be just as politically dangerous as listing too many. Note

that there are many other bodies that deal with internationalization;

however, none of them appear commonly in IETF standards work.




The International Organization for Standardization has been involved

with standards for scripts since before the IETF was started. ISO is a

non-governmental group made up of national bodies. ISO has many diverse

standards in the international scripts area; the one that is most used

in the IETF is commonly referred to as "ISO 10646", although its

official name has more qualifications. ISO 10646 describes a CCS that

covers almost all known written characters in use today.


ISO 10646 is controlled by the group known as "ISO/IEC JTC 1/SC 2 WG2",

often called "WG2" for short. ISO standards go through many steps before

being finished, and years often go by between changes to ISO 10646.

Information on WG2, and its work products, can be found at



Unicode Consortium


The second important group for international character standards is the

Unicode Consortium. The Unicode Consortium is a trade association of

companies and governments interested in promoting the Unicode Standard

[Unicode3]. The Unicode Standard is a CCS whose repertoire is identical

to ISO 10646. The Unicode Consortium has added features to the base CCS

which make it more useful in protocols, such as defining attributes for

each character.


The Unicode Consortium publishes addenda to the Unicode Standard as

Unicode Technical Reports. There are many types of technical reports at

various stages of maturity. The Unicode Standard and affiliated

technical reports can be found at <http://www.unicode.org/>


encodings and transformations of ISO 10646


Characters in the ISO 10646 CCS can be expressed in many ways. Encoding

forms are direct addressing methods, while transformation formats are

methods for expressing encoding forms as bits on the wire. Two encoding

forms are defined for ISO 10646:


UCS-2 addresses the first 2^15 characters as 16-bit values. This range

is also called the "Basic Multilingual Plane" (BMP), and is also called

"plane 0".


UCS-4 addresses the entire range of 2^31 characters as 32-bit values.


Many transformation formats of the CCS are used in IETF standards. The

two most common are:


UTF-8, defined in [RFC2279], is the preferred encoding for IETF

protocols. Characters in the BMP are encoded as one, two, or three



UTF-16 (BE & LE), defined in [RFC2781], is used much less often than

UTF-8. Characters in the BMP are always encoded as two octets, and many

characters outside the BMP as four octets.


native CCSs and charsets


Before ISO 10646 was developed, many countries developed their own CCSs

and charsets. Many dozen of these are in common use on the Internet

today. Examples include ISO 8859-5 (Russia) and Shift-JIS (Japan). The

official list of the registered charset tags is maintained by IANA.




Probably the most well-known native CCS is ASCII [US-ASCII]. This CCS is

used in numerous IETF protocols that have not yet been



local and regional standards organizations


Just as there are many native CCSs and charsets, there are many local

and regional standards organizations to create and support them. Common

examples of these are ANSI (United States), JIS (Japan), GB (China), and

CEN/ISSS (Europe).



4. Linguistic Issues


This section contains terms and topics that are commonly used in

linguistics and therefore are of concern to people internationalizing

protocols. These topics are standardized outside the IETF.


composition and decomposition


In some CCSs, some characters consist of combinations of other

characters. For example, the letter "a with acute" might be a

combination of the two characters "a" and "combining acute". The rules

for combining two or more characters are called "composition", and the

rules for taking apart a character into other characters is called



normalization and canonicalization


These two terms are often used interchangeably in internationalization.

Generally, they both mean to convert a string of one or more characters

into another string based on standardized rules. In internationalized

text, these rules are usually based on decomposing combined characters

or composing characters with combining characters. [UTR15] describes the

process and many forms of normalization in detail. Normalization is

important when comparing strings to see if they are the same.


locale and region


Because languages differ from country to country (and even region to

region within a country), the locale of the user of internationalized

text can often be an important factor. Typically, the locale information

for a user includes the language(s) used. Locale issues go beyond

character use, and can include things such as the display format for

currency, dates, and times.




In many scripts, particularly those from or based on European alphabets,

there are two forms for letters: uppercase and lowercase. There is

usually (but not always) a one-to-one mapping between the same letter in

the two cases. However, there are many examples of characters which

exist in one case but for which there is no corresponding character in

the other case. Case conversion can even be dependant on locale.

Converting between the two cases is sometimes called "folding".




Many processes have a need to order strings in a consistent sequence

(sorted). For some CCS/CES combinations, there is an obvious sort order

that can be done without reference to the linguistic meaning of the

characters: the codepoint order is sufficient. For other CCS/CES (such

as the ISO 2022 family) there is no such order that works well.


Codepoint order is usually not how any human educated by a local school

system expects to see strings ordered; if one orders to the expectations

of a human, one has a localized sort.


Sorting to codepoint order will seem inconsistent if the strings are not

normalized before sorting because different representations of the same

character will sort differently. This problem may be smaller with a

localized sort.




A graphic character or glyph is a character, other than a control

function, that has a visual representation (normally handwritten,

printed, or displayed). It is a recognizable abstract graphic symbol

which is independent of any specific design. There are many types of

glyphs, including alphabetic characters, digits, punctuation,

diacritics, and symbols.




A collection of glyph images having the same basic design.


code table or code page


A tabular representation of a coded character set, also showing the

coded representations.


code space


The numeric domain occupied by all bit combinations used for the coding

of a coded character set.


4.1 Types of characters




Characters from the spoken part of phonetic or syllabic scripts.

Examples include Latin letters a through z, Arabic letters, and Katakana

characters from Japanese.




Characters that, by themselves, represent words or concepts (as compared

to phonetic sounds). Examples include Han characters used in Chinese,

Japanese, and Korean.




Non-alphabetic characters that are used to delimit sounds, sentences,

and phrases. Examples include the period, comma, and hyphen.




Characters that represent pictures or icons (as compared to ones that

represent letters or punctuation). Examples include characters for

arrows, faces, and geometric shapes.


spacing characters


Characters that represent horizontal or vertical spaces in written text.

Examples include the space character, the tab character, and the

paragraph mark.




Characters that combine with alphabetic characters, usually to change

the spoken pronunciation of the base alphabetic character. Examples

include the combining acute accent, combining tilde, and combining ring



combining character


Characters that visually change the characters that precede them.

Examples include combining diacritics, many Arabic alphabetic letters,

and many letters from the Indic scripts.


control character


Non-displaying characters that cause changes in the systems in which

they are entered. Examples include the null character, the delete

character, and the bell character.


formatting character


Non-displaying character that has an effect on the surrounding

characters. Examples include characters for specifying the direction of

text, letter-spacing characters, and characters for specifying joining.



5. User interface for internationalized text


Although the IETF does not standardize user interfaces, many protocols

make assumptions about how a user will enter or see text that is used in

the protocol. Many protocols make inherent assumptions such as that text

will be typed on a standard (that is, US-centric) keyboard, or that text

will be displayed on a character-based monitor. Internationalization

challenges assumptions like these, and it is therefore useful to

consider how users typically interact with internationalized text.


input methods


Text can be entered into a computer in many ways. Keyboards are by far

the most common method used, but many characters cannot be entered on

typical computer keyboards. Many operating systems come with system

software that lets users input characters outside the range of what is

allowed by keyboards.


For example, there are dozens of different input method for Han

characters in Chinese, Japanese, and Korean. Some start with phonetic

input through the keyboard, while others use the number of strokes in

the character. Input methods are also needed for scripts that have many

diacritics, such as European characters that have two or three

diacritics on a single alphabetic character.


display methods


Many scripts can be directly displayed with fonts, where each character

from an input stream can simply be copied from a font system and put on

the screen. Other scripts need rules that are based on the input stream

in order to display text. Some examples of these display rules include:


- Scripts such as Arabic (and many others), where the form of the letter

changes depending on whether the letter is standing alone, at the

beginning of a word, in the middle of a word, or at the end of a word


- Scripts such as the Indic scripts, where consonants may change their

form if they are adjacent to certain other consonants


- Arabic and Hebrew script, where the order of the characters displayed

can be changed with right-to-left and left-to-right ordering marks


rendering characters


Combining characters modify the display of the character (or, in some

cases, characters) that precede them. When rendering such text, the

display engine must either find the character in the font that contains

the base character and all of the combining characters, or it must

render the combination itself. Such rendering can be straight-forward,

but it is sometimes complicated when the combining marks interact with

each other, such as when there are two combining marks that would appear

above one character.


bidirectional scripts


Most of the world's written languages are displayed left-to-right.

However, many widely-used written languages such as Hebrew and ones

based on the Arabic script are displayed right-to-left. Right-to-left

text often confounds protocol writers because they have to keep thinking

in terms of the order of characters in a string in memory, and that

order might be different than what they see on the screen.


Bidirectional text can cause even more confusion because there are

formatting characters in ISO 10646 which cause the order of display of

text to change. These formatting characters are needed so that one can

properly display right-to-left characters with left-to-right characters

at the same time without forcing the one of the strings to be stored in

reverse order. [Unicode3] has a long and incredibly detailed discussion

of bidirectional text.


undisplayable characters


Some characters have no displayable form. For instance, the zero-width

space (U+200B) cannot be displayed because it takes up no horizontal

space. Formatting characters such as those for setting the direction of

text are also undisplayable.



6. Text in current IETF protocols


Many IETF protocols started off being fully internationalized, while

others have been internationalized as they were revised. In this

process, IETF members have seen patterns in the way that many protocols

use text. This section describes some specific protocol interactions

with text.



protocol elements


Almost every protocol has named elements, such as "source port" in TCP.

In some protocols, the names of the elements

(or text aliases for the names) are transmitted within the protocol.

For example, in SMTP, the names of the verbs are part of the

command stream. The names of protocol elements are not normally seen

by end users.


on-the-wire encoding


Characters exist as codepoints in a charset. Before being transmitted

in a protocol, they must first be encoded. Similarly, when characters

are received in a transmission, they have been encoded, and a protocol

that needs to process the individual characters needs to decode them

before processing. The encoding and decoding used before and after

transmission is often called the "on-the-wire" (or sometimes just "wire")



name spaces


Many items in Internet protocols use names to identify content.

The field of names for particular item is called its name space.

The names in a name space may be controlled centrally (such as by IANA)

or may have distributed control, such as the names in the DNS.


parsed text


In some protocols, free text in text fields might be parsed. For

example, many mail user agents will parse the words in the text of

the Subject: field to attempt to thread based on the "Re:" prefix.


charset tagging


Protocols that allow more than one charset to be used on text that

is meant for human consumption usually require

that the text be tagged with the appropriate charset. Without this

tagging, a program looking at the text cannot definitively discern the

charset of the text.


language tagging


Some protocols allow text that is meant for machine processing

to be tagged with the language used in the text. Such tagging is important

for machine-processing of the text, such as by systems that "display"

the text by speaking it.




MIME (Multipurpose Internet Mail Extensions) is a message format that

allows for textual message bodies and headers in character sets other

than US-ASCII in formats that require ASCII (most notably, RFC 822).

MIME is described in RFCs 2045 through 2049.




Base64 is a transfer encoding syntax that allows binary data to be

represented by the ASCII characters A through Z, a through z, 0

through 9, +, /, and =. It is described in RFC 2045.


quoted printable


The original design of the quoted printable transfer encoding syntax was

to allow strings that had some non-ASCII printable characters mixed in

with mostly ASCII printable characters to be human readable. It is

described in RFC 2047. It is generally considered to be somewhat of a

failure at being readable.


ASN.1 text formats


The ASN.1 data description language has many formats for text items. The

formats allow for different repertoires and different encodings. Some of

the formats that appear in IETF standards based on ASN.1 include

IA5String (all ASCII characters), PrintableString (most ASCII

characters, but missing many punctuation characters), BMPString

(characters from ISO 10646 plane 0 in UTF-16BE format), UTF8String (just

as the name implies), and TeletexString (also called T61String; the

repertoire changes over time).


ASCII-compatible encoding


Starting in 2000, many ASCII-compatible encoding schemes (which are

actually transfer encoding syntaxes) have been proposed as possible

solutions for internationalizing host names. Their goal is to be able to

encode any string of ISO 10646 characters as legal DNS host names (as

described in STD 13). At the time of this writing, non ACE has become an

IETF standard.


7. Other Common Terms In Internationalization


This is a hodge-podge of other terms that have appeared in

internationalization discussions in the IETF. It is likely that

additional terms will be added as this document matures.




"Latin characters" is a not-precise term for characters derived from

Greek script and used throughout the world. The base Latin characters

make up the ASCII repertoire and have been augmented by many single and

multiple diacritics.




Because of the widespread use of Latin characters, people have tried to

represent many languages that are not based on a Latin repertoire in

Latin. This process is called "romanization". For example, there are two

popular romanizations of Chinese: Wade-Giles and Pinyin, the latter of

which is by far more common today. Most romanization systems are inexact

and do not give perfect round trip mappings between the native script

and the Latin characters.


CJK and Han


The ideographic characters used in Chinese, Japanese, Korean, and

traditional Vietnamese scripts are often called "CJK" characters after

the initial letters of the script names in English. They are also called

"Han" characters, after the romanized translation of the term in Chinese

that is often used for these characters. Note that CJK and Han

characters do not include the phonetic characters of the Japanese or

Korean alphabets.




The process of converting one language to another, or one script to

another, is called translation. Most language translation systems are

inexact and do not give one-to-one round trip mappings between the

languages. Most script translations are exact, and many have perfect

round-trip mappings.


regular expressions


Pattern matching for text involves being able to represent one or more

code points in an abstract notation, such as searching for all capital

Latin letters or all punctuation. The most common mechanism in IETF

protocols for naming such patterns is the use of regular expressions.


private use characters


Many CCSs have code points that are defined as existing characters that

have not defined semantics. The use of these "private use" characters is

defined by the parties who transmit and receive them, and is thus not

appropriate for standardization.



8. Security Considerations


Security is not discussed in this document.



9. References


[IDN-REQ] "Requirements of Internationalized Domain Names", work in

progress (draft-ietf-idn-requirements), Z. Wenzel and J. Seng.


[ISO10646] ISO/IEC 10646-1:2000. International Standard -- Information

technology -- Universal Multiple-Octet Coded Character Set (UCS) -- Part

1: Architecture and Basic Multilingual Plane.


[RFC1766] "Tags for the Identification of Languages", RFC 1766, H.



[RFC1766bis] "Tags for the Identification of Languages", work in

progress (draft-alvestrand-lang-tag-v2), H. Alvestrand.


[RFC2277] "IETF Policy on Character Sets and Languages", RFC 2277, H.



[RFC2279] "UTF-8, a transformation format of ISO 10646", RFC 2279, F.



[RFC2781] "UTF-16, an encoding of ISO 10646", RFC 2781,

P. Hoffman and F. Yergeau.


[Unicode3] The Unicode Consortium, "The Unicode Standard -- Version

3.0", ISBN 0-201-61633-5. Described at



[US-ASCII]  Coded Character Set -- 7-bit American Standard Code for

Information Interchange, ANSI X3.4-1986.


[UTR15] "Unicode Normalization Forms", Unicode Technical Report

#15, M. Davis & M. Duerst.



10. Additional Interesting Reading


ALA-LC Romanization Tables, Randall Barry (ed.), ISBN 0844409405


Blackwell Encyclopedia of Writing Systems, Florian Coulmas, ISBN



Unicode Standard version 3.0, Unicode Consortium, ISBN 0201616335


Writing Systems of the World, Akira Nakanishi, ISBN 0804816549



A. Acknowledgements


The definitions in this document come from many sources, including

a wide variety of IETF documents.


James Seng contributed to the initial outline of this document.

Others who contributed to the development include:


Jacob Palme

Susan Harris

Harald Alvestrand

Johan van Wingen

Yuri Demchenko



B. Author Contact Information


Paul Hoffman

Internet Mail Consortium and VPN Consortium

127 Segre Place

Santa Cruz, CA  95060 USA

paul.hoffman@imc.org and paul.hoffman@vpnc.org