From: Philippe Verdy (email@example.com)
Date: Wed Aug 18 2004 - 11:40:10 CDT
May be a fourth level of abstraction is needed to complete what the MIME
registry describes as "charsets": a TES (Transfer Encoding Syntax) sometimes
happen at end, and some legacy specifications of CES mix it with what should
have been left in a separate TES.
For example, the specification of SCSU (Simple Compression Scheme for
Unicode) defines it as a way to convert a stream of code points directly to
a stream of bytes, without going through the level of abstraction of
intermediate "code units" (or in this case, code units are simply the
This makes SCSU a legal CEF (like are UTF-32, UTF-16 and UTF-8) to convert a
stream of encoded characters into a stream of (8-bit) code units, and a
legal CES (like are UTF-32BE or UTF-32LE or UTF-16BE or UTF16-LE or UTF-8 or
CESU-8, and UTF-16 or UTF-32 or UTF-8 or CESU-8 with a leading BOM) to take
into account the generated byte order.
But the SCSU specification speaks about "optional extensions" which are
probably badly named because they should be better described as TES
(DLE-escaping for NUL and DLE, or run-length compression, or COBS encoding),
exactly like other well-known TES (Base64, Quoted-Printable) widely used in
I think that there still exists some other legacy charsets in the MIME
registry that mix these level of abstraction, where a clear separation
between CES and TES levels would have helped their interoperability. One
cause of this descrepancy is that it has been, since long easier to create a
new charset and have it registered in the long MIME registry, than to define
a clear TES separately (the TES registry in MIME is not extremely long, and
support for multiple TES in applications has often been very weak and not
easily extensible, developers prefering to develop first the support needed
to handle correctly the so many possible CES, just identified by their MIME
The other related "problem" of TES is that many document structures
(including XML) only offer a place to specify the "charset" (i.e. the result
of a combination of a CCS, CEF and CES), but no place to specify the TES,
which is left, apparently, to the transport protocol, ignoring the case of
local storage where identification of TES is nearly impossible to make
reliably... This means that local stores cannot benefit easily of the
advantages of a TES specification (for example, when creating a reference to
a text document, it's impossible to specify in the link that this document
has been COBS-encoded or Base-64 encoded or even compressed in deflate or
gzipped form, unless the local document is stored in an enveloppe format,
such as a RFC2118 message with headers, and there's support in the hyperlink
renderer to decode this enveloppe format transparently).
For now, a hyperlink can specify the MIME-type of the document with an
attribute specifying the "charset", i.e. the triplet <CCS,CES,CEF>, but no
reliable and documented attribute to specify its TES (unless the document is
transported via a email or with HTTP, and the source makes the job on the
fly to transform it to the desired TES, which is a CPU-intensive job for
servers that could be avoided if documents could be stored or cached
directly by the server in their TES-encoded form; this means support in the
server's storage to keep this out-of-band information).
There does exists solutions but they are not universal and interoperable
across distinct softwares working with the same physical document store:
some filesystems offer that support with out-of-band meta-data, some servers
will use private conventions with multiple file extensions and private
server configuration files...
If the document's TES encoding decoding could be handled directly by the
client, without dependance of the underlying transport or storage
technology, it would be easier.
TES encoding is really out of scope of Unicode, but its support in various
applications using encoded text documents should be enhanced. This includes
a support for it in the XML and HTML document syntax, notably within source
As a final note: multiple TES encoding stages may be chained in any
transport or storage, and changed on the fly across nodes in a transport
network, without affecting the charset used for the decoded document. But in
many applications, including HTTP, only one TES can be specified (else it
will break other features such as document content signature and
certification). I know no working implementation of any transport protocol
that transparently allows specifying these multiple TES encodings (most
often these steps are possible only in distinct layers of the transport
architecture, where it can be made transparent for the applications handling
encoded documents on the upper layers). This means that TES
encoding/decoding affects the performance (and reliability...) of each
relaying node in a transport network (such as proxies), a caveat avoided by
including TES within a MIME charset, so that no TES encoding (or more
precisely just a identity, do-nothing, "8-bit" TES encoding) will be
necessary in the relaying chain...
----- Original Message -----
From: "John Tisdale" <firstname.lastname@example.org>
Sent: Wednesday, August 18, 2004 5:27 AM
Subject: MSDN Article, Second Draft
Thanks everyone for your helpful feedback on the first draft of the MSDN
article. I couldn't fit in all of the suggestions as the Unicode portion is
only a small piece of my article. The following is the second draft based on
the corrections, additional information and resources provided.
Also, I would like to get feedback on the most accurate/appropriate term/s
for describing the CCS, CEF and CES (layers, levels, components, etc.)?
I am under a tight deadline and need to collect any final feedback rather
quickly before producing the final version.
Special thanks to Asmus for investing a lot of his time to help.
This archive was generated by hypermail 2.1.5 : Thu Aug 19 2004 - 15:28:11 CDT