Re: "ch" as in yecch

Date: Mon Oct 25 1999 - 00:04:23 EDT

       This is a valuable pair of contributions that has come out of a
       rambling and not very significant thread which has gone on
       mostly in an effort to help increase the general level of
       understanding of participants of this list as to how Unicode is
       used to represent in software the real-world elements of
       writing systems (not entirely successfully, I fear).

       Software developers, pay attention! (And thanks, Tex and Mark.)


       ---------------------- Forwarded by Peter
       Constable/IntlAdmin/WCT on 10/24/99 09:58 PM

       From: <> AT Internet on 10/23/99 04:14
             PM CDT

       Received on: 10/23/99

       To: Peter Constable/IntlAdmin/WCT, AT
       Subject: Re: "ch" as in yecch

       You raise a very good point.

       What we use in ICU and in Java is a BreakIterator, that gives
       you 'character'
       boundaries (you can also
       choose word, line and sentence boundaries). Cursor movement is
       one area where
       character boundaries are
       useful; searching is another (if a search matches, you want to
       ensure that the
       boundaries of the match are
       character boundaries.

       You use getCharacterInstance(Locale) to get the iterator; it
       will then give you
       boundaries on text. The
       character properties of the whole are extrapolated from the
       first code point.

       The Java interface is at:


       The ICU C++ interface is at


       It also has the overall documentation.

       The ICU C interface is in separate routines all starting with
       "ubrk_", e.g. to
       open, iterate, and close you
       would use the following:


       For random access, you use:
       owing.html, or


       Tex Texin wrote:

> Dear Uni-people,
> I am of course a supporter and a benefiter of Unicode
> and its many improvements over legacy encodings. As an
> implementer, and not a linguist, typographer, or nationalist
> not favoring one language or politics over another),
> I look to Unicode to provide me with standardization so I can
> provide world-wide plain-text support. I like that Unicode
> defines algorithms for bidirectional support, character
> and the like, and I am no longer in the business of
> both code pages and the algorithms for using those code
> (Well, I do a lot less of it now anyway. ;-) )
> As a software designer, I need to understand and rely on some
> basic principles. For example, I have to have a
> routine. This list can go on and on (and on...) about
> characters, versus letters, versus graphemes, but I need to
> implement something close to what a user expects, (or what I
> teach them to work with) and to have
> an element, or basic unit, that I can manipulate and design
       to use
> in my software.
> I can work with 16-bit units as Unicode defines them, and I
> program to either provide users with behaviors based on these
> units (e.g. cursor-right moves through each unit, i.e.
> each diacritic, tone mark, etc.) or I can
> provide users with a more complete element that users
       traditionally > think of as a character (e.g. cursor-right
       moves to the next letter).
> However, Unicode does not prescribe how I recognize or work
> with elements like "ch". As an implementer, I thought
> would look for a 16-bit abstract character (or multi-16-bit
> surrogate) followed by some non-spacing elements. Apparently,
> is not sufficient.
> I also thought I could look up properties of "characters" but
> don't know where to look up the properties of "ch".
> I imagine some of these "multi-non-spacing character"
> will cause me to redesign my approach to bidi as well.
> Although I recognize the many benefits of Unicode, if I
> understand how to reliably implement GetNextCharacter, and
> a property table for these "multi-non-spacing character"
> then many current designs for Unicode applications are now
> For developers, the greatest benefit of Unicode was it
> standardization for the basic character element.
> I suddenly feel thrown back into the multi-codepage
> quagmire of researching researching, researching, and
> continually revamping and re-generalizing my software to
       accomodate > new character types as I uncover them.
> I believe I do understand the rationales offered for why "ch"
> should not be a character. I would rather the onus be shifted
> input methods and legacy conversion programs to determine
> "ch" should be encoded as a single element or not, rather
> having all remaining software be continually analyzing this.
> I would not like for this note to kick off a repetition of
> that has already been said many times. Perhaps it is
> I would really like a clear definition for knowing how many
> or 16-bit words to read to get to the end of the current
> and how to look up the properties of that letter, its case,
> These are the basic elements I need to use in my software.
> The algorithms for doing this, should be able to support
       Slovak and > other languages in a uniform way.
> tex
> --
> Progress Software: The #1 Embedded Database
> Tex Texin Director, International
> Progress Software Corp. Voice: +1-781-280-4271
> 14 Oak Park Fax: +1-781-280-4949
> Bedford, MA 01730 USA

This archive was generated by hypermail 2.1.2 : Tue Jul 10 2001 - 17:20:54 EDT