i18n October 2008

i18n@lists.stg.fedoraproject.org

2 participants
7 discussions

Re: Pseudo-locales for i18n testing by English speakers
by Sean Flanigan 06 Oct '08

06 Oct '08

Martin Langhoff wrote: > 2008/10/2 Sean Flanigan <sflaniga(a)redhat.com>: >> I have a simple Ant task which can generate pseudo-translations like the >> one above from a gettext POT files, > > I am after a few sets of "latin-lookalike" character tables I can use. > Have you (or anyone) got pointers to good tables? Well, I've made up a couple of simple ones (also attached as UTF-8): ASCII: "abcdefghijklmnopqrstuvwxyz" BMP only: "åЬçđéϝցⱨîﺩⱪŀოňøÞᕴяšŧմⱱשẋŷż" BMP+SMP: "åЬçđ𝖾ϝցⱨî𝚓ⱪŀოňøÞᕴяšŧմⱱשẋŷż" You could also try googling for "LATIN SMALL LETTER {A,B,C,...} WITH", which should turn up all sorts of modified latin characters, such as LATIN SMALL LETTER V WITH RIGHT HOOK. Another option is the Wikipedia Unicode pages http://en.wikipedia.org/wiki/List_of_Unicode_characters has several sections for extended latin scripts, and the Unicode mapping tables down the bottom are handy if you want to go directly to a certain Unicode range (eg to get away from the BMP). > The simple example phrase you provided hit a bug in moodle (php > webapp) straight away - I think a few webapps have trouble with that > funny 'e' (U+1D5BE). Interestingly, it's also present in Jira > (Java-based webapp). Might be an iconv issue. I chose that 'e' specifically because it wasn't part of the BMP, but apparently the mathematical alphanumeric symbols are a bit of a special case - I'm not sure if systems are expected to provide font substitution for them. Zimbra (written in Java) had trouble with the 'e' too - it just removed it entirely. I think a lot of programs have trouble with characters that don't fit into 16-bit Unicode. My text editors and Thunderbird can show the 'e' character, but the cursor handling is all wrong on those lines. -- Sean Flanigan Senior Software Engineer Engineering - Internationalisation Red Hat ASCII: "abcdefghijklmnopqrstuvwxyz" BMP only: "åЬçđéϝցⱨîﺩⱪŀოňøÞᕴяšŧմⱱשẋŷż" BMP+SMP: "åЬçđ𝖾ϝցⱨî𝚓ⱪŀოňøÞᕴяšŧմⱱשẋŷż"

1 0

Re: Pseudo-locales for i18n testing by English speakers
by Asgeir Frimannsson 05 Oct '08

05 Oct '08

----- "Sean Flanigan" <sflaniga(a)redhat.com> wrote: > Ulrich Drepper wrote: > > Sean Flanigan wrote: > >> Would there be any interest in getting something like this into > glibc? > > > > Hell, no. There is no room for testing code in the runtime. > <snip> > Is the implementation of "fetch translations from MO files under > /usr/share/locale/" hard-coded? If there's already a nice > programmatic > hook I could use, even better. If I could register locale-specific > overrides of gettext(), I could add any number of dynamically > generated > locales. It is set by bindtextdomain(). Somewhat related, look at a previous discussion relating to Ubuntu's patched glibc for supporting language-packs: http://sources.redhat.com/ml/libc-alpha/2005-03/msg00105.html > A gettext() hook could also be used to fetch translations from other > sources, such as a shared, up-to-date, translation database. I think > that has the potential to be useful to a lot of people, not just > developers and testers. > > It doesn't really make sense to generate thousands of pseudo PO > files, > and compile them into static MOs, when all the required data (ie > English > text) is available at runtime. > > Let me change my question then. How would people feel about having a > hook to override the behaviour of gettext() in a system-wide fashion? I guess you could experiment with LD_PRELOAD for this. Tim Foster experimented with this a while a go. http://blogs.sun.com/timf/entry/how_much_translation_do_you cheers, asgeir

2 1

Re: Pseudo-locales for i18n testing by English speakers
by Sean Flanigan 02 Oct '08

02 Oct '08

Nicolas Mailhot wrote: >> But how can I find the name of the font which provides a given >> character? > > Answer in the fonts SIG wiki > http://fedoraproject.org/wiki/Category:Fonts_SIG > > Looking at your interests and questions, you should really read it and > join the SIG (same for other interested people) Thanks Nicolas, but I can't make much sense of that wiki. I gather that it covers the problem of "where can I find a font with character X?", which would be useful, but in this case I want to know "which of my current fonts is providing the character X?" Or when I choose the font "Monospace 10", which font *really* provides the character X? (Which probably depends on whether I'm using an X application, a Java Swing app or a Java SWT app...) I guess I should be reading up on font substitution. -- Sean Flanigan Senior Software Engineer Engineering - Internationalisation Red Hat

1 0

Re: Pseudo-locales for i18n testing by English speakers
by Sean Flanigan 02 Oct '08

02 Oct '08

Ulrich Drepper wrote: > Sean Flanigan wrote: >> Would there be any interest in getting something like this into glibc? > > Hell, no. There is no room for testing code in the runtime. I find it hard to believe that there is no testing code in the runtime. Even OS kernels have functions for debug messages. > And I see > absolutely no need whatsoever to have it there. It seems to be the > wrong place altogether. Perhaps it is; I'm mostly a Java programmer, but in Java I'd be looking to hook into the ResourceBundle class, which is responsible for fetching translated strings. I'd much prefer to keep my grubby mitts out of glibc, but I was given to understand that the gettext() calls at runtime are actually implemented in glibc (rather than say "libgettext"). If that's wrong, please enlighten me! > For PO files you create appropriate > translations. For the locales themselves you derive a file from the > existing locales, compile it using localedef, and just use it. > > None of the locale code is hardcoded in glibc. Why should this? Is the implementation of "fetch translations from MO files under /usr/share/locale/" hard-coded? If there's already a nice programmatic hook I could use, even better. If I could register locale-specific overrides of gettext(), I could add any number of dynamically generated locales. A gettext() hook could also be used to fetch translations from other sources, such as a shared, up-to-date, translation database. I think that has the potential to be useful to a lot of people, not just developers and testers. It doesn't really make sense to generate thousands of pseudo PO files, and compile them into static MOs, when all the required data (ie English text) is available at runtime. Let me change my question then. How would people feel about having a hook to override the behaviour of gettext() in a system-wide fashion? -- Sean Flanigan Senior Software Engineer Engineering - Internationalisation Red Hat

1 0

Re: Pseudo-locales for i18n testing by English speakers
by Sean Flanigan 02 Oct '08

02 Oct '08

Ding-Yi Chen wrote: > The pseudo locale is intriguing, and I assume it helps at some degree. > However, this approach does have its own limitation: Of course, pseudo-localisation testing is not the same as localisation testing in every Fedora language, but it's something! > 1. Lack of font support: as the attachment "lack_of_font.png" shows, the > pseudo locale might be rendered useless if all developers can see are > unicode boxes. :-P That tells me that the developers should install better fonts, or how else can they test an internationalised application? But to be honest, I probably shouldn't have used http://en.wikipedia.org/wiki/Mathematical_alphanumeric_symbols since they're only guaranteed to be available in certain mathematical fonts such as Code2001. I really need to find some latinesque characters that don't come from the BMP, nor from the maths section! Apparently Zimbra loses (without trace) the 'e' characters in my pseudotranslation. Bad Zimbra! As long as it's only a couple of characters, I think having some unusual characters is okay, since you can still work out what's going on, at least enough to resolve the problem by installing more fonts. > Perhaps we should specifiy the minimal font set as > remedy. Before running pseudo-localised apps, you mean? Good idea. I found a webapp that gives the names of unicode characters - <http://rishida.net/scripts/uniview/uniview.php>. Just paste text into the "cut & paste" field and hit enter. But how can I find the name of the font which provides a given character? I can tell you that all my pseudo-characters are readable on my computer, but I can't tell you where they come from. Once I work out what fonts my pseudo-locale requires, I'd be happy to share the info as a dependency list. Perhaps it would make sense to define a small Fedora package which specified certain Unicode fonts as dependencies, as well as enabling the hypothetical pseudo-locale support in glibc. > 2. It doesn't really solve the language specific problem. Take Chinese > characters sorting for example, they can be sorted by > Pinyin, Zhuyin, radical, number of strokes, and "natural" order such as > numberial characters. The sorting is impossible to verify without the > knowledge. True, but a pseudo-locale which uses reverse sorting can at least show up whether an app is using internationalised sorting, or plain old ASCII ordering. And we're not limited to what Microsoft did - I don't know much about Chinese character sorting, but we could probably come up with a couple of alternative sorts that could be understood by an English-speaking developer. But I don't want to tackle that just yet! > Still, the idea itself is good. And surely it filters out some of the > bugs without the help of translaters. I expect a lot of i18n/L10n bugs are not picked up until someone tests one of the affected languages. Some of those bugs could show up in a pseudo-locale much earlier, which has to be an improvement. For instance, I've already found bugs where Eclipse and joe mess up the cursor position when editing SMP characters, without personally knowing any SMP languages. As an English-only developer I think it's also pretty cool to see if my code is at least partly internationalised, which otherwise I can't see for myself at all, except in a foreign language. I think some English-only developers might take more interest in i18n issues if they could easily see the results for themselves. And for those i18n issues which can be demonstrated with a pseudo-locale, it can be easier for multiple developers to talk about something which is in "English", since most developers speak English, even if they have differing native languages. > Since the main purpose of pseudo locale is for testing, shall we agree > on a list of pseudo locales which have their own specified behaviour? I think it would be good if we could fit in with Vista's chosen pseudo-locale IDs, as listed here: http://blogs.msdn.com/shawnste/archive/2006/06/27/647915.aspx As I said, we certainly don't have to emulate MS completely, but I think we should use qps for the language code. See http://blogs.msdn.com/michkap/archive/2007/02/04/1596987.aspx As for the behaviours, I expect that they will change as we learn more from testing feedback, but here are some ideas: a. simple character substitution, rendered text to be about the same size b. character substitution with expansion (eg "[--- original text ---]") to make strings longer c. maybe swapping upper and lower case. Sometimes it's handy to have more than one pseudo-locale, eg to make sure a web client is not seeing the server's locale, so having spare locales might be handy. And we could have options like different sort orders. But I'd be happy to start with (a) or (b) and leave sort orders until a bit later. At least with (a) and (b) it's easy to see whether someone forgot to call gettext(), because the plain English strings will stick out. -- Sean Flanigan Senior Software Engineer Engineering - Internationalisation Red Hat

1 0

Pseudo-locales for i18n testing by English speakers
by Sean Flanigan 01 Oct '08

01 Oct '08

(Apologies for the double post on fedora-i18n-list, but I want to keep the thread together. Please reply to this message, not the first one.) G'day all, I think we should make use of pseudo-locales to test Fedora. [--- I ŧⱨîňⱪ ש𝖾 šⱨøմŀđ ოåⱪ𝖾 մš𝖾 øϝ Þš𝖾մđø-ŀøçåŀ𝖾š ŧø ŧ𝖾šŧ F𝖾đøяå. ---] (In case UTF-8 doesn't make it to everyone's mail client intact, the above sentence should like similar to the first one, except that the lower case characters have been replaced by other similar-looking Unicode characters. A couple of the characters don't fit into 16 bits, and really gave my text editors some trouble!) See http://en.wikipedia.org/wiki/Pseudo-translation and http://blogs.msdn.com/shawnste/archive/2006/06/27/647915.aspx for more about pseudo-locales. Microsoft actually used three different pseudo-locales to test Vista, with things like reverse sorting, right-to-left characters, and large character sets. To me, the main advantages of pseudo-localisation are the ability to test some aspects of i18n without having to wait for translations to be turned around, and allowing English-only speakers to test i18n areas, which is otherwise extremely difficult. I have a simple Ant task which can generate pseudo-translations like the one above from a gettext POT files, but I'm not suggesting that we should integrate my humble Ant task into the makefiles of thousands of Fedora packages. If the gettext runtime code that fetches translations from .mo files (in glibc?) were to recognise a pseudo-locale id, it could generate pseudo-translations on the fly from the English text. Admittedly, there's a little more to it than simple character substitution. The pseudo-translator has to avoid changing things like variable names and html tags, but a few rules (eg don't modify anything between angle/square/curly brackets, don't touch %d/%s/etc) would cover 95% of cases. In the other cases, you might mess up some HTML or fail to expand a variable, but only users who choose to use a pseudo-locale would ever see these problems. Would there be any interest in getting something like this into glibc? [--- S𝖾åň ---] PS this could make sense for the OpenJDK too, but that's another story. -- Sean Flanigan Senior Software Engineer Engineering - Internationalisation Red Hat

1 0

Pseudo-locales for i18n testing by English speakers
by Sean Flanigan 01 Oct '08

01 Oct '08

G'day all, I think we should make use of pseudo-locales to test Fedora. [--- I ŧⱨîňⱪ ש𝖾 šⱨøմŀđ ოåⱪ𝖾 մš𝖾 øϝ Þš𝖾մđø-ŀøçåŀ𝖾š ŧø ŧ𝖾šŧ F𝖾đøяå. ---] (In case UTF-8 doesn't make it to everyone's mail client intact, the above sentence should like similar to the first one, except that the lower case characters have been replaced by other similar-looking Unicode characters. A couple of the characters don't fit into 16 bits, and really gave my text editors some trouble!) See http://en.wikipedia.org/wiki/Pseudo-translation and http://blogs.msdn.com/shawnste/archive/2006/06/27/647915.aspx for more about pseudo-locales. Microsoft actually used three different pseudo-locales to test Vista, with things like reverse sorting, right-to-left characters, and large character sets. To me, the main advantages of pseudo-localisation are the ability to test some aspects of i18n without having to wait for translations to be turned around, and allowing English-only speakers to test i18n areas, which is otherwise extremely difficult. I have a simple Ant task which can generate pseudo-translations like the one above from a gettext POT files, but I'm not suggesting that we should integrate my humble Ant task into the makefiles of thousands of Fedora packages. If the gettext runtime code that fetches translations from .mo files (in glibc?) were to recognise a pseudo-locale id, it could generate pseudo-translations on the fly from the English text. Admittedly, there's a little more to it than simple character substitution. The pseudo-translator has to avoid changing things like variable names and html tags, but a few rules (eg don't modify anything between angle/square/curly brackets, don't touch %d/%s/etc) would cover 95% of cases. In the other cases, you might mess up some HTML or fail to expand a variable, but only users who choose to use a pseudo-locale would ever see these problems. Would there be any interest in getting something like this into glibc? [--- S𝖾åň ---] PS this could make sense for the OpenJDK too, but that's another story. -- Sean Flanigan Senior Software Engineer Engineering - Internationalisation Red Hat

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

i18n October 2008