On 06/22/2015 09:01 AM, Jonathan Dieter wrote:

On Mon, 2015-06-22 at 06:16 -0400, Jan Kurik wrote:

...

Recently we made it possible to install a small number of locales by supplying the rpm-macro “_install_langs”, for example

rpm -i -D _install_langs="en:de_DE" glibc-common.rpm

will install all English locales and all German locales which start with “de_DE”,

rpm -i -D _install_langs="en_US.utf8" glibc-common.rpm

will install only the en_US.utf8 locale,

rpm -i -D _install_langs="POSIX" glibc-common.rpm

will install nothing (but the POSIX/C is still available because it is builtin into glibc).

Please note that this step in implementing the feature will break deltarpms. Deltarpm requires the *full* data from the original rpm to be installed in order to build the updated rpm from the deltarpm, so the fact that we're only installing part of glibc-common will cause any deltarpms for glibc-common to fail.

If I understand correctly though, the above is a stepping-stone to subpackages containing the languages, which will *not* break deltarpms.

I do think the change is a great idea; I just want to make sure all involved are aware of this potential pitfall along the way.

It was known, but not mentioned, that this would decrease the effectiveness of deltarpm for glibc-common. This step is indeed just a stepping stone to splitting out the packages.

Cheers, Carlos.

On 06/22/2015 10:59 AM, Alexander Larsson wrote:

On mån, 2015-06-22 at 06:16 -0400, Jan Kurik wrote:

...

= Proposed System Wide Change: Glibc locale subpackaging = https://fedoraproject.org/wiki/Changes/Glibc_locale_subpackaging

Change owner(s):

• Mike Fabian <mfabian At redhat DOT com>
• Siddhesh Poyarekar <spoyarek AT redhat DOT com>
• Carlos O’Donell <codonell AT redhat DOT com>

This change should make it possible to install or uninstall locales individually.

== Detailed Description == Currently the file /usr/lib/locale/locale-archive contains all locales and is thus huge (103MB). For small systems (and containers) it would be useful to be able to install only a small number of locales. Recently we made it possible to install a small number of locales by supplying the rpm-macro “_install_langs”, for example

rpm -i -D _install_langs="en:de_DE" glibc-common.rpm

will install all English locales and all German locales which start with “de_DE”,

rpm -i -D _install_langs="en_US.utf8" glibc-common.rpm

will install only the en_US.utf8 locale,

rpm -i -D _install_langs="POSIX" glibc-common.rpm

will install nothing (but the POSIX/C is still available because it is builtin into glibc).

But this approach works only during an Anaconda based install when Anaconda supplies the _install_langs rpm-macro. When glibc is updated later, the _install_langs macro will not be supplied on the command line during the update and the default value “all” of “_install_langs” from /usr/lib/rpm/macros will be used and all locales come back during an update. Therefore, this solution is far from perfect. It should be made possible to install and uninstall locales individually, for example by having a separate package for the locales for each language. Installing such a package would add these locales to locale-archive, uninstalling it would remove them.

Do they really have to modify locale-archive? Can't each package install separate archive files (say, based on the locale name). Packaging optional extra files is a lot easier for me in my work with an xdg-app runtime based on fedora.

For now we are modifying locale-archive, but we are aware the this causes problems with container overlays, and are looking for a solution where this works on a per-file basis with each new subpackage langauge contributing a new loadable optimized binary locale file.

The stepping stone will likely be:

_install_langs (to limit languages) -> subpackages that modify locale-archive -> subpackages that install their own files

Please remember the core runtimes are key part of the OS, and we make these transitions slowly and gather metrics about how effective each step was and how many bugs we got, and if the intermediate changes broke anything.

In the end we may just skip the intermediate step depending on testing, but I won't guarantee that.

Cheers, Carlos.

Alexander Larsson wrote:

I understand that, and thanks for doing this work, its pretty important imho. I just wanted to make sure that you were aware of what the best end goal was. Btw, do you have any measures of the performance improvements of using the locale archive is in general? It seems a like many other distros chose not to use it.

As far as I know, they use it, but they build it in the equivalent of %post, using the tool upstream glibc provides for that, and with only the languages selected by the user. Fedora ships (at least it did before this Change) a prebuilt locale-archive with everything.

Kevin Kofler

Kevin Kofler wrote:

Carlos O'Donell wrote:

...

The stepping stone will likely be:

_install_langs (to limit languages) -> subpackages that modify locale-archive -> subpackages that install their own files

The funny thing is that your last step is probably the easiest to implement, because that's how subpackages are intended to work. Your first two steps require a lot of scriptlet hackery that you'll throw away if you implement the third.

PS: Separate per-locale files with proper %lang tagging would also make _install_langs just work without any scriptlet coding. And it would also work with a script like this (experimental and extremely slow proof of concept): http://svn.calcforge.org/viewvc/kannolo/trunk/kickstart/langpacks.sh?view=ma... (This script builds langpack tarballs for a whole live image. The idea is that one can then make a live image with minimal --instLangs and download and add the langpacks in a post-install step, before doing any package changes. But right now I am not using this script, it is just one of the ideas to have to make my Fedora Remix called Kannolo as small as possible in the future.)

Kevin Kofler

On Thu, 2015-08-27 at 16:10 +0200, Florian Festi wrote:

On 06/22/2015 12:16 PM, Jan Kurik wrote:

...

= Proposed System Wide Change: Glibc locale subpackaging =

We have to revisit this topic as soon as rich dependencies are in place. Rich dependencies offer a way to handle locales on a system wide level. One possible implementation would be having

langsupport-XX meta packages that enable support for a given language.

Locales could be in packages like glibc-lang-de, foo-lang-de, ... The main package could then have Requires: (foo-lang-de if langsupport-de) if we want to enforce the locale package to be installed.

Otherwise it might be more elegant to have the language package (e.g. glibc-lang-de) have:

Supplements: (glibc and langsupport-de)

As we can assume that glibc is always installed this could already be done with:

Supplements: langsupport-de

The benefit of this approach is that installing a new langsupport-XX package will bring in the locale packages for all packages making use of this mechanism.

Could that even be done automatically by rpmbuild?

Just like rpmbuild automatically generates a -debuginfo with the right files in it, could it generated the -lang-xx packages with the .mo files inside, and add the Supplements dependency you suggest?

Because doing it manually for all the languages supported by an application (which might vary from release to release) would be pretty awful. :-/

On Thu, Aug 27, 2015 at 05:53:19PM +0200, Florian Festi wrote:

In theory, yes. In the end we'd need something like find-lang.sh that actually spits out the packages to be included in the main spec file. debug-info packages are done that way. But it is not pretty.

It's not pretty behind the scenes, but packagers don't have to care. Being pretty *and* no one has to care seems ideal, but failing that, this is pretty good. :)