From: Mount, Rob (Robert F) (firstname.lastname@example.org)
Date: Thu Jun 05 2003 - 19:49:28 EDT
Thanks again for your response.
I understand what you say about word formation, and combining marks, and
that the Alphabetic
classification should not be limited to "L"s. But 30FC is of General
Category "Lm" (which should be
included) and, since version 3.1, is classified explicitly as Alphabetic in
(It appears that formal expression of the Alphabetic property was moved
to DerivedCoreProperties.txt in 3.1.) I don't understand why its exclusion
from the Alphabetic
category in 3.0.1 was not an oversight. But if not, then either the
consortium consensus on
the classification of this character has changed, or the current
classification is in error.
Here's a little more background regarding my motivation. The problem occurs
in a procedure
that evaluates whether a user-supplied name can be used as an identifier -
for which identification
of alphabetic characters is important. One implementation of isalpha(),
purportedly based on
Unicode 2.1, indicates that 30FC is an alpha character. The current
implementation from the
same vendor, based on 3.0.1, classifies it as non-alpha. Presumably the
next one will be based
on 3.1 or later and will reclassify it, again, as alpha.
I have since discovered section 5.16 of the spec which describes the Unicode
identifier formation, and frankly, our validation algorithm is a bit naive
and will require some
work. But our use of isalpha() is not, I think, fundamentally flawed; the
changes will require
only that we include some additional characters that are not currently
Certainly if the behavior of isalpha() did not change the existing algorithm
would at least
be stable across different platforms, warts and all. If we can't depend on
of isalpha() we will have to eliminate its use from our validation function.
So I am trying to discover why the behavior of isalpha() has changed. Here
possibilities: 1) the previous implementation was incorrect and the current
one is fixed;
2) the current implementation is flawed because it does not conform to the
standard; 3) the current implementation is flawed because it's based on
documentation of the standard; 4) both implementations are correct but are
different, incompatible standards; 5) something else I don't yet understand.
The overriding assumption for this entire discussion is that the behavior of
be governed by the Unicode Alphabetic property. That seems reasonable to me
and is, in
fact, the vendor's claim. If not, (or even if so) perhaps someone can
recommend a better
(or more stable) API for discovery of Unicode character metrics upon which
we might base
our identifier validation and other character processing logic.
From: Mark Davis [mailto:email@example.com]
Sent: Thursday, June 05, 2003 2:48 PM
To: Mount, Rob (Robert F); firstname.lastname@example.org
Subject: Re: Classification of U+30FC KATAKANA-HIRAGANA PROLONGED SOUND
Ah, I see why you didn't find the Alphabetic property. It was added in
Unicode 3.1.0 (March 2001), precisely to capture characters that are
not L yet are still alphabetic. If you look at the derivation in
C:\DATA\UCD\3.1.0-Update\DerivedCoreProperties-3.1.0.txt, you will
# Derived Property: Alphabetic
# Generated from: Lu+Ll+Lt+Lm+Lo+Nl + Other_Alphabetic
So Alphabetic includes all L's, but also other characters. And, as I
said, it alone is not sufficient for word breaks.
> Is the ommision of 30FC from the Alphabetic category of PropList.txt
This is not an oversight. As I said, many characters are not
Alphabetic and are still part of words. Examples include that
character and many others. As a simple case, "can't" is a word in
English, although the apostrophe is not alphabetic. There are many,
many examples using combining marks, such as a virama (halant) in
Hindi, which is not Alphabetic:
So if you want reasonable word-breaks, you need to use more than the L
category, you need to look at
? "Eppur si muove" ?
----- Original Message -----
From: "Mount, Rob (Robert F)" <email@example.com>
To: "Mark Davis" <firstname.lastname@example.org>; <email@example.com>
Sent: Thursday, June 05, 2003 11:57
Subject: RE: Classification of U+30FC KATAKANA-HIRAGANA PROLONGED
> Thanks to all who responded. The insight you provided is
invaluable. And I
> appreciate your patience with a UNICODE beginner.
> Mark's reference to UCD.html, and by inference to
> seems difinitive. However, these are part of the 4.0 spec. The
> implementation of isalpha is based, according to the vendor, on
> The vendor relys, instead, on
> which classifies 30FC as Diacritic, Extender, Bidi: Left-to-Right,
> Identifier Part, but not
> as Alphabetic. Is this an error in the specification? I could find
> reference to the Alphabetic
> property in the 3.0.1 documentation except in, and with reference
> However, it would seem, from the 4.0 documentation, that all
> having a General
> Category beginning with "L" should be considered as letters, and
> implicitly, as Alphabetic.
> Is this, indeed, the intent of the General Category classifications
> beginning with "L"?
> Is the ommision of 30FC from the Alphabetic category of PropList.txt
> -----Original Message-----
> From: Mark Davis [mailto:firstname.lastname@example.org]
> Sent: Thursday, June 05, 2003 9:28 AM
> To: Mount, Rob (Robert F); email@example.com
> Subject: Re: Classification of U+30FC KATAKANA-HIRAGANA PROLONGED
> The UCD has a property explicitly called "Alphabetic" in the UCD. So
> that should be used when determining whether a character is, well,
> alphabetic. See http://www.unicode.org/Public/UNIDATA/UCD.html
> However, in the past many people have misused functions like
> for doing more complicated processing like determining text
> (line and word breaks, for example). The function isAlpha() does not
> discriminate finely enough to be very accurate for processing like
> that. For more information, see
> Also see the proposed update to Unicode Regular Expressions, for
> discussion of the use of Unicode properties in connection with
> punct, etc. (in the context of regular expressions, at least).
> ? "Eppur si muove" ?
> ----- Original Message -----
> From: "Mount, Rob (Robert F)" <firstname.lastname@example.org>
> To: <email@example.com>
> Sent: Wednesday, June 04, 2003 16:11
> Subject: Classification of U+30FC KATAKANA-HIRAGANA PROLONGED SOUND
> > All,
> > I am investigating differing behavior in various environments of
> > wide-character version of the C function isAlpha with respect to
> > character U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK. Some
> > implementations indicate that it is alphabetic, some don't. I
> > suspect that other characters might be subject to the same
> > The UNICODE documents seem abiguous on this point: the General
> > Catetory is "Lm" which, although informative instead of normative,
> > would seem to imply that it is alphabetic; likewise
> > DerivedCoreProperties-4.0.0.txt indicates that it is alphabetic;
> > PropList-4.0.0.txt contains two records - one indicating that it
> > a diacritic, one that indicates it is an extender.
> > On to my questions:
> > Q1: Can a character be both alphabetic and diacritic?
> > Q2: Is there a difinitive answer as to whether this is an
> > character?
> > Thanks in advance for answers to these questions and/or any
> > additional isight you can provide.
> > Regards,
> > Rob Mount
This archive was generated by hypermail 2.1.5 : Thu Jun 05 2003 - 20:42:16 EDT