From: Jim Monty (firstname.lastname@example.org)
Date: Wed Nov 03 2010 - 17:20:44 CST
Doug Ewell wrote:
> Jim Monty <jim dot monty at yahoo dot com> wrote:
> > Is there a utility, preferably open source and written in C, that inspects
> > UTF-16/UTF-16BE/UTF-16LE text and identifies broken surrogate pairs and
> > characters? Ideally, the utility can both report illegal code units and
> > them by replacing them with U+FFFD.
> What's an "illegal" character, for purposes of this exercise? Do you
> mean a noncharacter, or something else?
I mean the sixty-six code point Unicode reserves as noncharacters (e.g.,
But I'm most keenly interested in a utility that detects broken UTF-16 surrogate
pairs. By this, I mean a 16-bit code unit in the range from 0xD800 thru 0xDBFF
not immediately followed by a 16-bit code unit in the range from 0xDC00 thru
0xDFFF, and vice versa.
I need to repair broken UTF-16 text that some software (e.g., GNU iconv) and
programming languages (e.g., Perl) choke on. See this discussion of the topic on
This archive was generated by hypermail 2.1.5 : Wed Nov 03 2010 - 17:42:05 CST