L2/14-195

Subject: Data on the usage of left-side spacing marks in New Tai Lue
Source: Roozbeh Pournader, Google Inc.
Date: August 5, 2014


In L2/14-090, Martin Hosken suggested various approaches to change the model
of the encoding of New Tai Lue script in Unicode. The script subcommittee
recommended also had recommendations about this in L2/14-129. UTC discussed the
topic in meeting #139, but didn't arrive at a decision.

The author tried to gather information based on the usage of New Tai Lue on the
web. The whole Google web corpus of public web pages was investigated. The
research was limited to HTML pages that didn't have any malformed or unassigned
Unicode text. Non-HTML documents and HTML documents that had unpaired surrogates
or unassigned Unicode characters were not considered.

The research confirmed suspicions that the visual "wrong" encoding of New Tai
Lue (Hosken's Solution 3), is much more widespread that the logical "correct"
Unicode encoding (Hosken's solution 1).

This can be verified by searching for perhaps the most frequent New Tai Lue word
that exposes the encoding issue, ᦙᦵᦲᧂ (<U+1999, 19B5, 19B2, 19C2>, <LETTER
LOW MA, VOWEL SIGN E, VOWEL SIGN II, LETTER FINAL NG), pronounced /mə᷆ŋ/ and
meaning "city" or "country".

Visual encoding (E, MA, II, NG):
http://www.google.com/search?q=%E1%A6%B5%E1%A6%99%E1%A6%B2%E1%A7%82&filter=0

Logical encoding (MA, E, II, NG):
http://www.google.com/search?q=%E1%A6%99%E1%A6%B5%E1%A6%B2%E1%A7%82&filter=0

As of August 5, 2014, the first page reports "About 8970 results", while the
second reports "About 115".

It should be noted that the corpus of publicly available New Tai Lue web pages
is extremely small. The author did not find the time to study copying among
different web pages thoroughly, but a quick glance at the above search results
above shows that the numbers are not that meaningful:

The visual encoding results is basically used by three domain names, dw12.com,
tai12.com, and taitham.com, which may all be the same website (they look
visually very similar, if not the same).

The logical encoding is basically from the definition of the word on wiktionary
at http://en.wiktionary.org/wiki/%E1%A6%99%E1%A6%B5%E1%A6%B2%E1%A7%82, with its
sister websites and their various copies around the web.

It should also be noted that there are other potential encoding issues with
New Tai Lue. For example, it appears that the vowel sign AE may have been
represented more as as sequence of <19B5, 19B5> instead of the standard <19B6>.

The author doesn't wish to draw conclusions from this information, and wishes to
note that Microsoft products have been supporting logical New Tai Lue since
at least October 2009, and Google and Mozilla products (and various other
products based on HarfBuzz) have done the same since at least February 2013. The
amount of private data or public non-HTML data in the correct encoding is
unknown to the author.