Target Audience: Software Engineer

Level of Session: Intermediate, Advanced

For many operations on text, knowing the language of the text can improve results. Invoking the right spell checker, language specific layout module, or text-to-speech module are examples of this. Markup languages like HTML and rich text formats like RTF provide means to 'tag' all parts of the text with a language designator. These features are not always used, and other data formats such as plain text, don't provide information on the language. When it would be useful to know the language of a piece of text, but language tags are not available, the language of the text can be guessed with surprising accuracy from even short runs of text. There are several common techniques that can be used to guess language.

When the world wants to talk, it speaks Unicode
Program Showcase Past Conferences Registration Unicode Standard Call for Papers
Accommodation Sponsors Talks and Papers Travel Conference Board Next Conference
International Unicode Conferences are organized by Global Meeting Services, Inc., (GMS). GMS is pleased to be able to offer the International Unicode Conferences under an exclusive license granted by the Unicode Consortium. All responsibility for conference finances and operations is borne by GMS. The independent conference board serves solely at the pleasure of GMS and is composed of volunteers active in Unicode and in international software development. All inquiries regarding International Unicode Conferences should be addressed to info@global-conference.com.

Unicode and the Unicode logo are registered trademarks of Unicode, Inc. Used with permission.

5 July 1999, Webmaster