On Tue, Apr 1, 2008 at 7:36 PM, Philippe Verdy <verdy_p@wanadoo.fr> wrote:

Rick McGowan wrote:
> Yesterday afternoon I received a report that the Unicode.org
> server was so slow that ordinary HTML pages were timing out.
>
> I found hundreds of suspicious open connections for IP address
> 41.232.132.25 repeatedly downloading one huge file.

These repeated connections may be the result of repeated slow repsonse from
the server. This may be a bug on a tool used on that Egyptian site in Cairo
(but it may be caused also by repeated loose of sessions caused by its
Egyptian ISP, when the user just tries to get the same file, or could be
caused by a bogous downloader tool that makes excessive reppeated requests
in case of server-side failure or connection loss.

> I blocked that IP address, and this resolved the problem.
>
> If you find that the server becomes much slower than usual,
> please feel free to report this to me and I will investigate.

This explains why so many attempts to connect to the CLDR Survey failed.

Or took minutes to validate each vote, or just display each page.

I wanted to post a comment to the CLDR forum and my comment was lost as well
after several attempts to resubmit it (no response from server).

Anyway, the CLDR Survey is still very slow, and this has not changed a lot
since the last survey period last year. I do think it is taking too much
resources, or there's something wrong in the database design, and it takes
too much time trying to parse and reparse again the same XML documents or
regenerating them to get them in form. The XML generation for the
publication should be probably be done by extracting from a more traditional
RDBMS (with tables, fields and indexes) rather than with large XML
documents, if this is waht it is doing.

I don't understand why this is much slower to conduct the survey with very
few users connected, than driving a classic web site with cheap server
components like PHP and MySQL over Apache: all those websites can easily
manage lots of simultaneous users, and still perform complex scripting, such
as running a Wiki engine.

I know that the Survey is written with server-side Java components, but
certainly, it suffers a lot from very slow performance. I don't know which
server you use (Oracle?), but it seems that the code generating the HTML
pages from the survey application really would need some revamping. Also the
XML database updates when submitting data seems to create extremely complex
transactions to update lots of things, possibly in many shared XML
documents, and this apparently requires regenerating the complete set of XML
files, instead of just updating a few fields; this causes lots of contention
due to exclusive locks, and no more than a handful of users can decently
work conencted in the Survey app.

Each action requires lot of patience, and we also frequently loose the HTTP
session, having to logon again and revalidating the same screen: the session
timeout is apparently very short, and given the response time of the server,
about half of my attempts to update things fail. It's also completely
impossible to work with it at some hours.

The process is then extremely discouraging, and we need lots of
perseverance. The bad thing is that this discouragement will have
consequences: not enough vetters to accept the changes, and even the UTC
members with higher privileges in the Survey don't come to update their
choice (this was seen in the last survey period, and even now, they have
apparently still not connected to the Survey since its recent reopening:
this is visible in the number of submitted votes per item, where I am still
alone in my French locale...)

So the problem is not that the CLDR Survey is slower than usual, it has
always been very slow!
Given the incresed number of items to validate, with a single category
containing more than 2100 items per locale, i.e. 27 pages containing 80
items, that require about 15 minutes of work per page, due to long reposne
time and loss of sessions, reviewing this single category will require about
8 hours of continuous work for this category. Reviewing the complete set
will require more then 20 hours of continuous work, i.e. about 3 weeks with
a reasonnable time of not more than one hour per day on average (and if I
connect to it every day, something that is not reasonnable).

So we are really too far away in needed work time, and too near from the
time-limit of the submission period (which is about one month in length).
Last year I wanted to militate so that the Survey subission period would be
continuous (permantently opened), with just the review period being limited,
by working on a snapshot taken for a beta release.

One solution would be to put the submission process on a separate server,
using more traditional solutions: an opened catalaog of open-source XML
files, and a versioning system like CVS. IT could also be hosted else where
than on the Unicode site, using larger platforms made to host hundreds of
open-sourced projects; the issue would be related to the copyright and
licencing assignment required which may be incompatible with the CLDR
copyright and licencing policy.

Also many vetters are not concerned by all the CLDR data categories. It
should be possible to extract several subschemas from the CLDR, and work
directly on them with a simpler system that will be more efficient.

More works is needed also to reorganize some very large categories, notably
the local time metazones. Having to navigate in categories with hundreds or
thousands of entries that are not directly related to the same schema level
is really a nightmare. IT's hard to consolidate and unify the presentations
and possible errors are more difficult to locate. No category should contain
more than about 240 entries (3 pages), and then it should be possible to
present all the items of the same category on the same page, so that we can
make sure to get a coherent and complete set of usable data (incoherences
and incompleteness of most categories makes the whole CLDR database unusable
directly for any project without having to review it privately again before
using it, so this cannot help creating a standard, and migrations from one
version of the CLDR database to another is unlikely to occur easily if this
additional review is too lengthy).

When I look at the various entries, it's clear that each category has a
structure that is currently not used as it should: there are essential
entries, and many optional entries which take their values from default
aliasings inherited from Root. These additional entries should be on
separate subcategories. On the opposite, there are main categories that
contain just a handfull of entries. The contrast is evident: the Survey tool
was not designed to work on categories at various levels of the XML data
tree.

So the problem is then not solved. I still experience frequent loss of
sessions and reviewing the same data again on large pages makes the progress
really slow to perform, and it's probable that all other contributors
experience the same repeated problems. The lack of reviewers has a
consequence: the CLDR project is still not viable as a working effective
standard for collaboration, and every site just consult it sometime to build
their own locale data sets with their own uncorrected errors (as
demonstrated on the Google search web site for example).