Intelligent Data Processing Using Unicode: Japanese Address "Data Cleansing"

Tammy Young - Basis Technology Corporation

Intended Audience: Manager, Software Engineer, Marketer
Session Level: Beginner

Trillium Software (a division of Harte-Hanks Data Technologies) is a leader in its field of "data cleansing" sometimes associated with "data mining." Trillium's data cleansing software annually saves its customers millions of dollars by helping phone companies, financial institutions, or mail order houses merge together disparate, mammoth databases by standardizing fields across databases, extraction of important information buried in free-form text fields, and removing duplicate records. Intelligence in Trillium can surmise that "Mary E. Smith" and "M. E. Smith" and "Mary Smith" living at the same address are the same person and thus eliminate fuzzy duplicates. When Trillium Software decided to take Trillium to Japan, they called on Basis Technology for its expertise in Unicode-enablement and cultural/linguistic expertise in Asia.

Trillium is an excellent case study of using Unicode to allow Trillium to process Japanese data. In this project, we modified an application to process database information using Unicode as its internal encoding, while allowing for multiple Japanese legacy encodings on input and output. In the first part of the project, we dealt with several Unicode-related issues (1) creating mapping tables between Japanese mainframe vendor encodings and Unicode, (2) ensuring roundtrip convertibility between encodings, and (3) data storage and sorting. The second half involves creating an all-Unicode Japanese address and name parser with the intelligence to parse free-form text fields into the name and address elements. Though the areas covered are specific to Trillium, issues resolved can also be generalized to other Unicode-enablement situations.

When the world wants to talk, it speaks Unicode
Unicode Standard Program Conference Board WWW9 Talks and Papers Past Conferences
Showcase Registration Accommodation Travel Sponsors Next Conference
International Unicode Conferences are organized by Global Meeting Services, Inc., (GMS). GMS is pleased to be able to offer the International Unicode Conferences under an exclusive license granted by the Unicode Consortium. All responsibility for conference finances and operations is borne by GMS. The independent conference board serves solely at the pleasure of GMS and is composed of volunteers active in Unicode and in international software development. All inquiries regarding International Unicode Conferences should be addressed to info@global-conference.com.

Unicode and the Unicode logo are registered trademarks of Unicode, Inc. Used with permission.

16 March 2000, Webmaster