From: Philippe Verdy (email@example.com)
Date: Wed Nov 05 2003 - 08:35:48 EST
From: "Peter Kirk" <firstname.lastname@example.org>
> Agreed. But to be fair to MySQL, they do mention as a potential problem
> that three bytes have to be allocated in strings for each UTF-8
> character. For full UTF-8 support they would need four bytes per
> character which would, from their perspective, be a greater problem.
> Also I suspect that Unicode data is actually being stored in 16-bit
> entities, and that the major issue is the extra complication of handling
> surrogate pairs within that representation (rather than the trivial one
> of converting such pairs to and from valid UTF-8).
Modern database engines now offer multiple encoding strategy for storage of
characters. In SQL engines, the key issue is performance (notibly in terms
of storage I/O or networking I/O), but this is completely orthogonal of the
logical correctness of SQL functions and selections, which should be based
internally on Unicode characters, independantly of their actual encoding in
storage (as UTF-8, CESU-8, UTF-16BE/LE, UTF-32, GB18030, or any other legacy
So I do think that it is quite easy to implement UTF-8 and be fully
compliant with it for I/O in the request language or in its results, as well
as for storage where it is certainly better than CESU-8.
The hard part is not in these interfaces (MySQL for example is unique in the
fact that it supports several alternate storage formats for its tables), but
in the core engine itself when it performs identity selection, sorting and
range selections, substring extractions.
The other part of the problem is the interoperability with MySQL clients. As
long as these clients will not be prepared to receive character data out of
the BMP, they should connect with a CESU-8 encoding profile. If they are
prepared for it, they should better use UTF-8. But is the MySQL client
protocol compatible enough to support explicit tagging of the encoding used
for strings? This is the good question. This may require an update in the
protocol, and this may not be the first priority for MySQL, which wants
first to prepare its core engines, and get it to connect to external data
sources or storages like Oracle, Sybase, MS-SQL, UTF-8 text files, Access
MDB files, XML data files, and possibly with more recent extensions of the
Berkeley DB table format that now supports characters out of the BMP.
Tracking the required reencoding between these components connected to the
core engine may be tricky to develop, unless the interfaces between these
components and the engine are prepared to support explicit labelling of the
charsets and encodings actually usable to interoperate, and some negociation
protocol in these interfaces (something like the "Accept-*" headers in
HTTP), notably if there are transcoding issues (which may affect very
serious database integrity constraints, notably for uniqueness and
existence, but also in triggers).
This archive was generated by hypermail 2.1.5 : Wed Nov 05 2003 - 09:21:36 EST