Adding Surrogate Support to a Unicode Library

Markus Scherer - Unicode Technology Group of IBM in Cupertino, CA

Intended Audience: Software Engineer, Systems Analyst
Session Level: Intermediate

This talk discusses the need to support "surrogate characters", analyzes many of the implementation choices with their pros and cons, and presents a practical example.

As the preparation of the second part of ISO 10646 and the next version of Unicode draws to an end, Unicode applications need to prepare to support assigned characters outside the BMP. Although the Unicode encoding range was formally extended via the "surrogate" mechanism with Unicode 2.0 in 1996, many implementations still assume that a code point fits into 16 bits. At the end of this year, the drafts for the new standard versions are expected to be stable, and the assignment of surrogate characters for use in the East Asian markets will soon require implementations that support the full Unicode range.

For example, the International Components for Unicode (ICU), an open-source project, provides low-level Unicode support in C and C++ similar to the Java JDK 1.1. In order to support the full encoding range, some of the APIs and implementations had to be changed. Several alternatives were discussed for this project and are presented in this talk:

- Moving from UCS-2 to either UTF-16, UTF-32, or UTF-8
- Using code points, pairs of surrogates, or strings for single characters
- Writing generic code that works with any of the UTFs

The ICU APIs and implementation are now being adapted for UTF-16, with 32-bit code point values for single characters, and the lookup of character properties is extended to work with surrogate characters. This approach is compared with what other companies and organizations are doing, especially for Java, Linux and other Unixes, and Windows.

When the world wants to talk, it speaks Unicode
Unicode Standard Program Conference Board Call for Papers Talks and Papers Past Conferences
Showcase Registration Accommodation Travel Sponsors Next Conference
International Unicode Conferences are organized by Global Meeting Services, Inc., (GMS). GMS is pleased to be able to offer the International Unicode Conferences under an exclusive license granted by the Unicode Consortium. All responsibility for conference finances and operations is borne by GMS. The independent conference board serves solely at the pleasure of GMS and is composed of volunteers active in Unicode and in international software development. All inquiries regarding International Unicode Conferences should be addressed to info@global-conference.com.

Unicode and the Unicode logo are registered trademarks of Unicode, Inc. Used with permission.

19 Jun 2000, Webmaster