Network Working GroupA. Phillips, Ed.
Internet-DraftwebMethods, Inc.
Expires: July 12, 2005M. Davis
 January 11, 2005

Matching Language Identifiers


Status of this Memo

This document is an Internet-Draft and is subject to all provisions of section 3 of RFC 3667. By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she become aware will be disclosed, in accordance with RFC 3668.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on July 12, 2005.

Copyright Notice

Copyright (C) The Internet Society (2005).


This document describes different mechanisms for comparing and matching the language identifiers defined by RFC3066bis. Possible algorithms for language negotiation and content selection are described. Portions of this document obsolete RFC 3066.Phillips, A., Ed. and M. Davis, Ed., Tags for the Identification of Languages (Internet-Draft), January 2005.[1]

Table of Contents

1.  Introduction
2.  The Language Range
    2.1  Extended Language Range
    2.2  Meaning of the Language Tag
        2.2.1  Default Matching Scheme
    2.3  Other Matching Schemes
    2.4  Considerations for Private Use Subtags
3.  IANA Considerations
4.  Security Considerations
5.  Character Set Considerations
6.  References
§  Authors' Addresses
A.  Acknowledgements
§  Intellectual Property and Copyright Statements


1. Introduction

Human beings on our planet have, past and present, used a number of languages. There are many reasons why one would want to identify the language used when presenting or requesting information.

Information about a user's language preferences commonly needs to be identified so that appropriate processing can be applied. For example, the user's language preferences in a browser can be used to select web pages appropriately. A choice of language preference can also be used to select among tools (such as dictionaries) to assist in the processing or understanding of content in different languages.

Given a set of language identifiers, such as those defined in RFC3066bis, various mechanisms can be envisioned for performing language negotiation and tag matching. The suitability of a particular mechanism to a particular application depends on the needs of that application.

This document defines language ranges and syntax for specifying user preferences in a request for language content. It also specifies a default algorithm for matching language ranges to content (language tags), as well as alternate mechanisms suitable for certain applications.

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119]Bradner, S., Key words for use in RFCs to Indicate Requirement Levels, March 1997.[12].


2. The Language Range

A Language Range is a set of languages whose tags all begin with the same sequence of subtags. A Language Range can be represented by a 'language-range' tag, by using the definition from HTTP/1.1Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P. and T. Berners-Lee, Hypertext Transfer Protocol -- HTTP/1.1, June 1999.[17] :

language-range = language-tag / "*"

That is, a language-range has the same syntax as a language-tag or is the single character "*". This definition of language-range implies that there is a semantic relationship between tags that share the same subtag prefixes.

A language-range matches a language-tag if it exactly equals the tag, or if it exactly equals a prefix of the tag such that the first character following the prefix is "-". (That is, the language-range "en-de" matches the language tag "en-DE-boont", but not the language tag "en-Deva".)

The special range "*" matches any tag. A protocol which uses language ranges may specify additional rules about the semantics of "*"; for instance, HTTP/1.1 specifies that the range "*" matches only languages not matched by any other range within an "Accept-Language:" header.

As noted above, not all languages or content denoted by a specific language-range may be mutually intelligible and this use of a prefix matching rule does not imply that language tags are assigned to languages in such a way that it is always true that if a user understands a language with a certain tag, then this user will also understand all languages with tags for which this tag is a prefix. The prefix rule simply allows the use of prefix tags if this is the case.

2.1 Extended Language Range

The simple matching described above is not always the most appropriate use of the information contained in language tags. Some applications may wish to define a more granular matching scheme based on extended language ranges:

extended-language-range = (subtag / "*") *("-" (subtag / "*"))
subtag = (1*8alphanum)
alphanum = ALPHA / DIGIT

In this language range scheme, a language range takes the form of a series of subtags or the special subrange of "*". For example, the language range "en-*-US" specifies a primary language of 'en', followed by any script subtag, followed by the region subtag 'US'.

Ed.Note> A more exact ABNF is possible to construct. It would need to capture the structure of langtag and show that each field is optional. My first stab at it was something like:

(primary_lang / "*")
[[[ "-" (script / "*")]
 [[ "-" (region / "*")]
  [ ("-*" / *("-" variant))]]]]

2.2 Meaning of the Language Tag

The language tag always defines a language as spoken (or written, signed or otherwise signaled) by human beings for communication of information to other human beings.

If a language tag B contains language tag A as a prefix, then B is typically "narrower" or "more specific" than A. For example, "zh-Hant-TW" is more specific than "zh-Hant".

This relationship is not guaranteed in all cases: specifically, languages that begin with the same sequence of subtags are NOT guaranteed to be mutually intelligible, although they may be. For example, the tag "az" shares a prefix with both "az-Latn" (Azerbaijani written using the Latin script) and "az-Cyrl" (Azerbaijani written using the Cyrillic script). A person fluent in one script may not be able to read the other, even though the text might be identical. Content tagged as "az" most probably is written in just one script and thus might not be intelligible to a reader familiar with the other script.

The relationship between the tag and the information it relates to is defined by the standard describing the context in which it appears. Accordingly, this section can only give possible examples of its usage.

2.2.1 Default Matching Scheme

Implementations that are searching for content or otherwise matching language tags to a language-range [Section 2The Language Range] may choose to assume that there is a semantic relationship between two tags that share common prefixes. This is called 'language tag fallback'. The most common implementation follows this pattern:

When searching for content using language tag fallback, the language tag is progressively truncated from the end until a match is located. For example, starting with the tag "en-US-boont", searches or matches would first be performed with the whole tag, then with "en-US", and finally with "en". This allows some flexibility in finding content. It also typically provides better results when data is not available at a specific level of tag granularity or is sparsely populated (than if the default language for the system or content were used).

Tag to match: en-US-boont
1. en-US-boont
2. en-US
3. en

 Figure 4: Default Fallback Pattern Example 

When working with tags and ranges you should also note the following:

  1. Private-use and Extension subtags are normally orthogonal to language tag fallback. Implementations should ignore unrecognized private-use and extension subtags when performing language tag fallback. Since these subtags are always at the end of the sequence of subtags, they naturally fall out of the default fallback pattern (above). Thus a request to match the tag "en-US-boont-x-1943" would produce exactly the same information content as the example above.
  2. Implementations that choose not to interpret one or more private-use or extension subtags should not remove or modify these extensions in content that they are processing. When a language tag instance is to be used in a specific, known protocol, and is not being passed through to other protocols, language tags may be filtered to remove subtags and extensions that are not supported by that protocol. This should be done with caution, since it it is removing information that may be relevant if services on the other end of the protocol would make use of that information.
  3. Some applications of language tags may want or need to consider extensions and private-use subtags when matching tags. If extensions and private-use subtags are included in a matching process that utilizes the default fallback mechanism, then the implementation should canonicalize the language tags and/or ranges before performing the matching. Note that language tag processors that claim to be "well-formed" processors as defined in [1]Phillips, A., Ed. and M. Davis, Ed., Tags for the Identification of Languages (Internet-Draft), January 2005. generally fall into this category.

2.3 Other Matching Schemes

Implementations MAY choose to implement different styles of matching for different kinds of processing. For example, an implementation could treat an absent script subtag as a "wildcard" field; thus "az-AZ" would match "az-AZ", "az-Cyrl-AZ", "az-Latn-AZ", etc. but not "az". If one item is to be chosen, the implementation could pick among those matches based on other information, such as the most likely script used in the language/region in question.

Because the primary language subtag cannot be absent, the 'UND' subtag might sometimes be used as a 'wildcard' for this style of matching. For example, in a query where you want to select all language tags that contain 'Latn' as the script code and 'AZ' as the region code, you could use "und-Latn-AZ".

Extended language ranges are designed around this idea. An extended language range matches a tag if:

Each subtag in the extended language range that is not "*" exactly matches the subtag in the language tag in that position. For example, the range "en-*-US" matches "en-Latn-US".

Each subtag in the extended language range that is "*" has a corresponding subtag in the language tag or that subtag is empty. For example, the range "en-*-US" matches "en-Latn-US" and also "en-US".

Each subtag type that is not specified in the extended language range may contain additional values. For example, the range "en-*-US" matches the tag "en-Latn-US-boont".

Implementations may also wish to use semantic information external to the langauge tags when performing fallback. For example, the primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal Norwegian) might both be usefully matched to the more general subtag 'no' (Norwegian). Or an application might infer that content labeled "zh-CN" is morely likely to match the range "zh-Hans" than equivalent content labeled "zh-TW".

2.4 Considerations for Private Use Subtags

Private-use subtags require private agreement between the parties that intend to use or exchange language tags that use them and great caution should be used in employing them in content or protocols intended for general use. Private-use subtags are simply useless for information exchange without prior arrangement.

The value and semantic meaning of private-use tags and of the subtags used within such a language tag are not defined. Matching private use tags using language ranges or extended language ranges may result in unpredictable content being returned.


3. IANA Considerations

This document presents no new or existing considerations for IANA.


4. Security Considerations

The only security issue that has been raised with language tags since the publication of RFC 1766, which stated that "Security issues are believed to be irrelevant to this memo", is a concern with language ranges used in content negotiation - that they may be used to infer the nationality of the sender, and thus identify potential targets for surveillance.

This is a special case of the general problem that anything you send is visible to the receiving party. It is useful to be aware that such concerns can exist in some cases.

The evaluation of the exact magnitude of the threat, and any possible countermeasures, is left to each application protocol.

Although the specification of valid subtags for an extension MUST be available over the Internet, implementations SHOULD NOT mechanically depend on it being always accessible, to prevent denial-of-service attacks.


5. Character Set Considerations

The syntax in this document requires that language ranges use only the characters A-Z, a-z, 0-9, and HYPHEN-MINUS legal in language tags. These characters are present in most character sets, so presentation of language tags should not have any character set issues.

Rendering of characters based on the content of a language tag is not addressed in this memo. Historically, some languages have relied on the use of specific character sets or other information in order to infer how a specific character should be rendered (notably this applies to language and culture specific variations of Han ideographs as used in Japanese, Chinese, and Korean). When language tags are applied to spans of text, rendering engines may use that information in deciding which font to use in the absence of other information, particularly where languages with distinct writing traditions use the same characters.


6 References

[1] Phillips, A., Ed. and M. Davis, Ed., "Tags for the Identification of Languages (Internet-Draft)", January 2005.
[2] International Organization for Standardization, "ISO 639-1:2002, Codes for the representation of names of languages -- Part 1: Alpha-2 code", ISO Standard 639, 2002.
[3] International Organization for Standardization, "ISO 639-2:1998 - Codes for the representation of names of languages -- Part 2: Alpha-3 code - edition 1", August 1988.
[4] ISO TC46/WG3, "ISO 15924:2003 (E/F) - Codes for the representation of names of scripts", January 2004.
[5] International Organization for Standardization, "Codes for the representation of names of countries, 3rd edition", ISO Standard 3166, August 1988.
[6] Statistical Division, United Nations, "Standard Country or Area Codes for Statistical Use", UN Standard Country or Area Codes for Statistical Use, Revision 4 (United Nations publication, Sales No. 98.XVII.9, June 1999.
[7] ISO 639 Joint Advisory Committee, "ISO 639 Joint Advisory Committee: Working principles for ISO 639 maintenance", March 2000.
[8] Hardcastle-Kille, S., "Mapping between X.400(1988) / ISO 10021 and RFC 822", RFC 1327, May 1992.
[9] Borenstein, N. and N. Freed, "MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies", RFC 1521, September 1993.
[10] Alvestrand, H., "Tags for the Identification of Languages", RFC 1766, March 1995.
[11] Hovey, R. and S. Bradner, "The Organizations Involved in the IETF Standards Process", BCP 11, RFC 2028, October 1996 (HTML, XML).
[12] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997 (HTML, XML).
[13] Freed, N. and K. Moore, "MIME Parameter Value and Encoded Word Extensions: Character Sets, Languages, and Continuations", RFC 2231, November 1997 (HTML, XML).
[14] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, November 1997.
[15] Berners-Lee, T., Fielding, R. and L. Masinter, "Uniform Resource Identifiers (URI): Generic Syntax", RFC 2396, August 1998 (HTML, XML).
[16] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998 (HTML, XML).
[17] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999 (HTML, XML).
[18] Carpenter, B., Baker, F. and M. Roberts, "Memorandum of Understanding Concerning the Technical Work of the Internet Assigned Numbers Authority", RFC 2860, June 2000.
[19] Alvestrand, H., "Tags for the Identification of Languages", BCP 47, RFC 3066, January 2001.
[20] Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003.
[21] Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, July 2002.


Authors' Addresses

  Addison Phillips (editor)
  webMethods, Inc.
  432 Lakeside Drive
  Sunnyvale, CA 94088
EMail:  aphillips@webmethods.com
  Mark Davis
EMail:  mark.davis@us.ibm.com


Appendix A. Acknowledgements

Any list of contributors is bound to be incomplete; please regard the following as only a selection from the group of people who have contributed to make this document what it is today.

The contributors to RFC 3066 and RFC 1766, the precursors of this document, made enormous contributions directly or indirectly to this document and are generally responsible for the success of language tags.

The following people (in alphabetical order) contributed to this document or to RFCs 1766 and 3066:

Glenn Adams, Harald Tveit Alvestrand, Tim Berners-Lee, Marc Blanchet, Nathaniel Borenstein, Eric Brunner, Sean M. Burke, Jeremy Carroll, John Clews, Jim Conklin, Peter Constable, John Cowan, Mark Crispin, Dave Crocker, Martin Duerst, Michael Everson, Doug Ewell, Ned Freed, Tim Goodwin, Dirk-Willem van Gulik, Marion Gunn, Joel Halpren, Elliotte Rusty Harold, Paul Hoffman, Richard Ishida, Olle Jarnefors, Kent Karlsson, John Klensin, Alain LaBonte, Eric Mader, Keith Moore, Chris Newman, Masataka Ohta, George Rhoten, Markus Scherer, Keld Jorn Simonsen, Thierry Sourbier, Otto Stolz, Tex Texin, Andrea Vine, Rhys Weatherley, Misha Wolf, Francois Yergeau and many, many others.

Very special thanks must go to Harald Tveit Alvestrand, who originated RFCs 1766 and 3066, and without whom this document would not have been possible. Special thanks must go to Michael Everson, who has served as language tag reviewer for almost the complete period since the publication of RFC 1766. Special thanks to Doug Ewell, for his production of the first complete subtag registry, and his work in producing a test parser for verifying language tags.


Intellectual Property Statement

Disclaimer of Validity

Copyright Statement