Re: UTF-16 is not Unicode

From: David Starner (starner@okstate.edu)
Date: Tue Feb 12 2002 - 11:15:24 EST


On Tue, Feb 12, 2002 at 11:22:01AM +0100, Marco Cimarosti wrote:
> At best, the localization could use a label such as "Unicode (UTF-8)" to
> enforce the concept that UTF-8 is Unicode as well. But it could hardly use
> "Unicode (UTF-16BE)" for the *default* UTF, because the user would ask
> "Where is *plain* 'Unicode'?"

Then possibly the user should be educated. This gets hairer if you think
about crossplatform; UTF-16 is not suitable for native use on Unix
systems (the whole reason UTF-8 was created), so the user should be
encouraged to use UTF-8 for the default Unicode encoding on those
platforms.
 
> A ideal interface should probably automatically and silently select Unicode
> (and its default UTF) whenever one or more of the characters in a document
> are not representable in the local encoding.

NOOOOO! Don't ever save in an encoding that isn't the local encoding
without user interaction. You can very quickly get mojibake with that -
the next program will probably open the file in the local encoding. It
also tends to annoy the user when the program picks the encoding,
ignoring what everything else on the system uses. Offer to change the
encoding or tranliterate.
 
> When the user selects "International (Unicode)", he should be allowed to
> enter an "Advanced settings" menu which, for this encoding, allows choosing
> between "8 bit ASCII-compatible (UTF-8)", "16 bit with surrogates support
> (UTF-16)", "flat 32 bit (UTF-32)". Selecting "16 bit ... (UTF-16)" shows
> extra choices like "Big-endian" vs. "Little Endian".

That's too painful, unless you're doing something like Netscape that has
to provide a million encoding options. UTF-32 is worthless as a disk
encoding. Handle both endians of UTF-16 correctly, and assume everyone
else does - if they don't, then a converter program can be found and
used. That reduces your choices to two: UTF-8 and UTF-16. If you must
complicate things, SCSU is a more useful complication.
 

-- 
David Starner / Давид Старнзр - starner@okstate.edu
Pointless website: http://dvdeug.dhis.org
What we've got is a blue-light special on truth. It's the hottest thing 
with the youth. -- Information Society, "Peace and Love, Inc."



This archive was generated by hypermail 2.1.2 : Tue Feb 12 2002 - 10:52:55 EST