Re: Case blind comparison

From: Kenneth Whistler (
Date: Tue Jul 29 1997 - 18:46:19 EDT

> In The Unicode Standard, Version 2.0, section 4.1, it states that
> "Because there are many more lowercase forms than there are uppercase
> or titlecase, it is recommended that the lowercase form be used for
> normalization, such as when strings are case-folded for loose
> comparison or indexing." It appears to me that the uppercase version
> should be used for exactly that reason. Can anyone explain to
> me the advantage of mapping to lower case in general?

The situation is as follows. There are many instances of lowercase
Latin letters which do not have an uppercase form. There are
no instances of uppercase Latin letters which do not have a lowercase
form encoded as a character in Unicode.

If you normalize to lowercase, then it will in general be true
that forall c in s islower(c) is TRUE. Whereas, if you normalize
to uppercase, then the corresponding statement is not true, i.e.
(forall c in s isupper(s) is TRUE) is FALSE (or TRUE, depending).

If you look at the history of Latin typography, in the more distant
past it used to be that the majuscule was the unmarked form.
("unmarked" and "marked" here are used in their structuralist sense.)
The miniscule forms were introduced in calligraphy and formed
the basis for the lowercase in typography when case distinctions
started to become a standard part of Western European language

But in the modern era, the situation has reversed. It is the
lowercase forms which are the unmarked forms of the letters.
This has been deeply influenced by IPA, which greatly extended
the range of lowercase baseform letters, without introducing
corresponding uppercase letters (in principle, since case
distinctions are irrelevant in phonetic transcription). When
IPA was used as the basis for other standard Latin-based
orthographies, especially in Africa, uppercase letters corresponding
to some of the lowercase forms started to be invented, so that
uppercase/lowercase orthographical conventions could be
carried across to the new orthographies.

I suspect that the engineering practice of normalizing to
uppercase derives from the days of 5-bit and 6-bit codes.
In 5-bit code (remember telegrams printed by teletypes?),
there were *only* uppercase letters A-Z. If you are normalizing
between a 5-bit code and a 6-bit code, you must uppercase,
because that is the only case in common. This practice of
normalizing to uppercase became a convention that could be
carried harmlessly into 7-bit and 8-bit codes, since they
always contained case pairs for the added letters. It basically
was a harmless choice to go either way, and established
convention ruled.

Normalizing to uppercase should, however, be rethought now in
the context of the Universal Character Set, where case pairs
for all letters are not automatically available, and where
normalizing to the unmarked case (lowercase) is the preferable

--Ken Whistler

This archive was generated by hypermail 2.1.2 : Tue Jul 10 2001 - 17:20:36 EDT