Re: Detecting encoding in Plain text

From: John Delacour (JD@BD8.COM)
Date: Thu Jan 08 2004 - 07:49:27 EST

  • Next message: "Re: Detecting encoding in Plain text"

    At 12:09 pm +0000 8/1/04, wrote:

    >There is no foolproof way of differentiating between some of the
    >encodings. While UTF-16 or UTF-8 with a BOM (such files don't
    >necessarily start with a BOM by the way) "stand out" as being
    >unlikely to be in any other encoding others are more troublesome.

    Given any sizeable chunk of text, it ought to be possible to estimate
    the statistical likelihood of its being in a certain
    encoding/[language] even if it's in an unspecified 8859-* encoding.
    It would be quite an interesting exercise, but I'd be surprised if
    someone hasn't done it before. Perhaps someone here knows.


    This archive was generated by hypermail 2.1.5 : Thu Jan 08 2004 - 08:31:24 EST