Re: Feedback on the proposal to change U+FFFD generation when decoding ill-formed UTF-8

From: Alastair Houghton via Unicode <>
Date: Wed, 31 May 2017 19:11:59 +0100

> On 30 May 2017, at 18:11, Shawn Steele via Unicode <> wrote:
>> Which is to completely reverse the current recommendation in Unicode 9.0. While I agree that this might help you fending off a bug report, it would create chances for bug reports for Ruby, Python3, many if not all Web browsers,...
> & Windows & .Net
> Changing the behavior of the Windows / .Net SDK is a non-starter.
>> Essentially, "overlong" is a word like "dragon" or "ghost": Everybody knows what it means, but everybody knows they don't exist.
> Yes, this is trying to improve the language for a scenario that CANNOT HAPPEN. We're trying to optimize a case for data that implementations should never encounter. It is sort of exactly like optimizing for the case where your data input is actually a dragon and not UTF-8 text.
> Since it is illegal, then the "at least 1 FFFD but as many as you want to emit (or just fail)" is fine.

And *that* is what the specification says. The whole problem here is that someone elevated one choice to the status of “best practice”, and it’s a choice that some of us don’t think *should* be considered best practice.

Perhaps “best practice” should simply be altered to say that you *clearly document* your behaviour in the case of invalid UTF-8 sequences, and that code should not rely on the number of U+FFFDs generated, rather than suggesting a behaviour?

Kind regards,


Received on Wed May 31 2017 - 13:12:21 CDT

This archive was generated by hypermail 2.2.0 : Wed May 31 2017 - 13:12:21 CDT