Re: detecting encoding in plain text (related to utf8)

From: Markus Scherer (
Date: Wed Jan 14 2004 - 12:17:48 EST

  • Next message: Mark Davis: "Re: Detecting encoding in Plain text"

    Deepak Chand Rathore wrote:
    > unicode range
    > utf 8 encoded bytes
    > U-00000000 - U-0000007F: 0xxxxxxx
    > U-00000080 - U-000007FF: 110xxxxx 10xxxxxx
    > U-00000800 - U-0000FFFF: 1110xxxx 10xxxxxx 10xxxxxx
    > ...

    This table is not correct. Please check Unicode 3.2 or Unicode 4 for the correct table.

    Table 3.1B. Legal UTF-8 Byte Sequences in
    Conformance chapter in

    > But, there is one concern. In some cases the utf8 byte stream starts with a
    > BOM,( for eg. when we try reading bytes from a text file that
    > is saved using notepad (using utf8 option )in WIN2k, after first few bytes(
    > i suppose first 3 bytes), the actual text start.
    > So how do we detect whether the byte stream starts with a BOM or not ??
    > or the first few bytes represent BOM or the actual text ??

    There is a whole FAQ section on this topic at

    Best regards,

    Opinions expressed here may not reflect my company's positions unless otherwise noted.

    This archive was generated by hypermail 2.1.5 : Wed Jan 14 2004 - 12:57:54 EST