Is there better documentation on how multilib works and how to resolve multilib problems than http://fedoraproject.org/wiki/PackagingDrafts/MultilibTricks ? I understand that it uses various heuristics before it corrupts files, but those don't seem to be documented.
I just spent half a day tracking down a problem which turned out to be because multilib had corrupted a file, and I wish to solve this now once and for all.
Rich.
Richard W.M. Jones (rjones@redhat.com) said:
Is there better documentation on how multilib works and how to resolve multilib problems than http://fedoraproject.org/wiki/PackagingDrafts/MultilibTricks ? I understand that it uses various heuristics before it corrupts files, but those don't seem to be documented.
I just spent half a day tracking down a problem which turned out to be because multilib had corrupted a file, and I wish to solve this now once and for all.
'corrupted a file'? I'm really not sure what you mean here.
In any case, excessive verbiage follows:
...
Fedora, on certain architectures, supports multible binary ABIs. This includes support for i386 on x86_64, and ppc64 on ppc.
Fedora supports these ABIs through a process called 'multi-arch' or 'multilib', wherein the corresponding RPM packages for the secondary arch are included along with their primary arch counterparts. These packages are built via the same process as the primary arch packages, and are identical to the packages that would be shipped in a release where that arch would be the primary arch.
The primary function of multilib support in Fedora is to provide a runtime environment for applications to use. To satisfy this, packages that provide runtime libraries are included for the secondary arch.
The secondary function of multilib support in Fedora is to provide a development environment. To satisfy this, packages that provide development headers are included for the secondary arch. However, it should be noted that it is preferred to use a full system of the appropriate architecture to do development where possible; this can be done by using the 'mock' package to create an appropriate chroot.
Multilib packages are intentionally not provided for system daemons (for example, to run alternate-wordsize versions of Apache or PostgreSQL.) If you need to run a non-default wordsize of a system daemon, it is recommended that you install the system with a release that supports that architecture as the primary architecture.
Not all packages produced by a particular source RPM are included as multilib packages; only subpackages that implement runtime libraries or development environments are included.
Whether or not multilib packages are installed by default is controlled by the 'multilib_policy' configuration file parameter for yum. The default value for Fedora of 'best' installs only packages for the primary architecture. If this is changed to 'all' packages for all compatible architectures will be installed by default. Note that this can only be changed post-install; anaconda uses the default of only installing packages for the primary arch.
You can still at any time install packages for a particular arch by passing the architecture directly to yum. For example, running 'yum install glibc.i686', or putting 'glibc.i686' in a kickstart %packages stanza, will install the 32-bit x86 version of glibc.
When installed in parallel, multilib RPM packages may contain files that live in common paths. When this happens, RPM uses the following algorithm:
- If the file in both packages is identical, installation is allowed and the file is written - If the file in both packages is an ELF binary, the file used is the file in the package for the primary architecture - If the file in both packages is not an ELF binary a RPM conflict is raised
If such conflicts are raised with packages in the Fedora tree, it can be considered a packaging bug that needs to be fixed.
...
Bill
On Mon, Jan 25, 2010 at 02:08:10PM -0500, Bill Nottingham wrote:
- If the file in both packages is identical, installation is allowed and the file is written
- If the file in both packages is an ELF binary, the file used is the file in the package for the primary architecture
- If the file in both packages is not an ELF binary a RPM conflict is raised
Are you sure about the third rule?
I think the problem arises because /usr/bin/libguestfs-supermin-helper is different in libguestfs.i686 and libguestfs.x86_64, but this file is a shell script, not an ELF binary.
These are the files in libguestfs.i686 and libguestfs.x86_64 that have the same names but different content:
/usr/bin/hivexget (ELF binary) /usr/bin/hivexml (ELF binary) /usr/bin/libguestfs-test-tool (ELF binary) /usr/libexec/libguestfs-test-tool-helper (ELF binary) /usr/bin/libguestfs-supermin-helper (shell script)
Anyway, I will move the shell script and see if that helps.
Rich.
On Mon, 25 Jan 2010, Richard W.M. Jones wrote:
On Mon, Jan 25, 2010 at 02:08:10PM -0500, Bill Nottingham wrote:
- If the file in both packages is identical, installation is allowed and the file is written
- If the file in both packages is an ELF binary, the file used is the file in the package for the primary architecture
- If the file in both packages is not an ELF binary a RPM conflict is raised
Are you sure about the third rule?
Yes.
I think the problem arises because /usr/bin/libguestfs-supermin-helper is different in libguestfs.i686 and libguestfs.x86_64, but this file is a shell script, not an ELF binary.
These are the files in libguestfs.i686 and libguestfs.x86_64 that have the same names but different content:
/usr/bin/hivexget (ELF binary) /usr/bin/hivexml (ELF binary) /usr/bin/libguestfs-test-tool (ELF binary) /usr/libexec/libguestfs-test-tool-helper (ELF binary) /usr/bin/libguestfs-supermin-helper (shell script)
Anyway, I will move the shell script and see if that helps.
Wont help, the multilib file "shadowing" logic is NOT based on paths. Looking at libguestfs.spec, here's the problem:
%global _use_internal_dependency_generator 0
The external dependency generator doesn't create the file "coloring" that the multilib magic needs to operate. So you'll get conflicts on all paths where content differs.
- Panu -
On Tue, 26 Jan 2010, Panu Matilainen wrote:
On Mon, 25 Jan 2010, Richard W.M. Jones wrote:
On Mon, Jan 25, 2010 at 02:08:10PM -0500, Bill Nottingham wrote:
- If the file in both packages is identical, installation is allowed and the file is written
- If the file in both packages is an ELF binary, the file used is the file in the package for the primary architecture
- If the file in both packages is not an ELF binary a RPM conflict is raised
Are you sure about the third rule?
Yes.
I think the problem arises because /usr/bin/libguestfs-supermin-helper is different in libguestfs.i686 and libguestfs.x86_64, but this file is a shell script, not an ELF binary.
These are the files in libguestfs.i686 and libguestfs.x86_64 that have the same names but different content:
/usr/bin/hivexget (ELF binary) /usr/bin/hivexml (ELF binary) /usr/bin/libguestfs-test-tool (ELF binary) /usr/libexec/libguestfs-test-tool-helper (ELF binary) /usr/bin/libguestfs-supermin-helper (shell script)
Anyway, I will move the shell script and see if that helps.
Wont help, the multilib file "shadowing" logic is NOT based on paths. Looking at libguestfs.spec, here's the problem:
%global _use_internal_dependency_generator 0
The external dependency generator doesn't create the file "coloring" that the multilib magic needs to operate. So you'll get conflicts on all paths where content differs.
...except looking at libguestfs-1.0.75-1.fc12, it *has* been built with internal dependency generator, and it gives:
Preparing... ########################################### [100%] file /usr/bin/libguestfs-supermin-helper conflicts between attempted installs of libguestfs-1:1.0.75-1.fc12.i686 and libguestfs-1:1.0.75-1.fc12.x86_64
...which is the expected behavior for a non-elf file (script in this case but doesn't matter). If both arch's of libguestfs have been installed, --replacefiles or --force has been used to force the install despite of a conflicting file.
- Panu -
On Tue, Jan 26, 2010 at 12:35:19PM +0200, Panu Matilainen wrote:
...except looking at libguestfs-1.0.75-1.fc12, it *has* been built with internal dependency generator, and it gives:
Preparing... ########################################### [100%] file /usr/bin/libguestfs-supermin-helper conflicts between attempted installs of libguestfs-1:1.0.75-1.fc12.i686 and libguestfs-1:1.0.75-1.fc12.x86_64
The problem seems to be when you install them in separate RPM transactions, then remove one (or possibly upgrade one). I haven't worked out a reliable reproducer, but I do know it can happen on machines where nothing has been forced.
Anyway, I rewrote the script so that it's the same on all architectures now.
Rich.
On Tue, 26 Jan 2010, Richard W.M. Jones wrote:
On Tue, Jan 26, 2010 at 12:35:19PM +0200, Panu Matilainen wrote:
...except looking at libguestfs-1.0.75-1.fc12, it *has* been built with internal dependency generator, and it gives:
Preparing... ########################################### [100%] file /usr/bin/libguestfs-supermin-helper conflicts between attempted installs of libguestfs-1:1.0.75-1.fc12.i686 and libguestfs-1:1.0.75-1.fc12.x86_64
The problem seems to be when you install them in separate RPM transactions, then remove one (or possibly upgrade one). I haven't worked out a reliable reproducer, but I do know it can happen on machines where nothing has been forced.
It shouldn't be possible to install packages with conflicting files, separate transactions or not, without some sort of forcing. If it is, then its a rather grave bug somewhere in rpm file conflict logic and needs finding and fixing. A reproducer would be very much appreciated.
Note that rpm < 4.6.0 did behave differently here: packages with conflicting files were allowed to be installed in the same transaction but not if installed separately, leading to strange situations. So if there's a chance you've tested on RHEL/Centos at some point, that could explain it as it'd then be just a matter of which order the packages got installed in.
- Panu -
On Tue, Jan 26, 2010 at 06:43:35PM +0200, Panu Matilainen wrote:
Note that rpm < 4.6.0 did behave differently here: packages with conflicting files were allowed to be installed in the same transaction but not if installed separately, leading to strange situations. So if there's a chance you've tested on RHEL/Centos at some point, that could explain it as it'd then be just a matter of which order the packages got installed in.
That could well explain why we've only seen this reported on EPEL 5. I thought it was a coincidence, but looks like it could be the older RPM version there.
Rich.
On Tue, 26 Jan 2010, Richard W.M. Jones wrote:
On Tue, Jan 26, 2010 at 06:43:35PM +0200, Panu Matilainen wrote:
Note that rpm < 4.6.0 did behave differently here: packages with conflicting files were allowed to be installed in the same transaction but not if installed separately, leading to strange situations. So if there's a chance you've tested on RHEL/Centos at some point, that could explain it as it'd then be just a matter of which order the packages got installed in.
That could well explain why we've only seen this reported on EPEL 5. I thought it was a coincidence, but looks like it could be the older RPM version there.
Okay, if it has only been seen on EPEL then no need to look further. IIRC it got fixed in RHEL 5 too, 5.3 or thereabouts.
- Panu -
On Tue, Jan 26, 2010 at 12:13:33PM +0200, Panu Matilainen wrote:
%global _use_internal_dependency_generator 0
The external dependency generator doesn't create the file "coloring" that the multilib magic needs to operate. So you'll get conflicts on all paths where content differs.
How can one both get the file "coloring" and add own dependencies with a script to an rpm? Iirc the internal dependency generator needs to be disabled to be able to specify a special script to add dependencies.
Regards Till
On Tue, 26 Jan 2010, Till Maas wrote:
On Tue, Jan 26, 2010 at 12:13:33PM +0200, Panu Matilainen wrote:
%global _use_internal_dependency_generator 0
The external dependency generator doesn't create the file "coloring" that the multilib magic needs to operate. So you'll get conflicts on all paths where content differs.
How can one both get the file "coloring" and add own dependencies with a script to an rpm? Iirc the internal dependency generator needs to be disabled to be able to specify a special script to add dependencies.
The sorry state of things right now is that there's no generic way that works in every situation with the internal dependency generator, but for some cases its possible:
1) All the require/provide helper scripts the internal dep generator uses can be overridden and customized, such as the __perl_provides/requires scripts. 2) Custom requires etc can be added in a scripted manner with Requires: %(some-custom-script) but this is rather limited as the script would run at early spec parsing state, where one typically wants to generate dependencies from what was just compiled. Which is possible (but cumbersome) for the things that use helper scriptlets, ie case 1)
FWIW, I'm working on it. The internal dependency generator is a serious bottleneck to what packagers would want/need to do, and the pile of ugly hacks in it is getting a bit much.
The first step is probably to move all the actual dependency generation work out of librpmbuild into helpers that can be overridden and customized, so that 1) becomes a general (even if cumbersome) solution. The main offender is elf dependency extraction which needs to be lifted into separate helper, which should also open new possibilities for dealing with non-native elf files (ie cross-builds).
It needs a whole lot more than that of course but you gotta start somewhere...
- Panu -
On Fri, Jan 29, 2010 at 12:07:35AM +0200, Panu Matilainen wrote:
On Tue, 26 Jan 2010, Till Maas wrote:
On Tue, Jan 26, 2010 at 12:13:33PM +0200, Panu Matilainen wrote:
%global _use_internal_dependency_generator 0
The external dependency generator doesn't create the file "coloring" that the multilib magic needs to operate. So you'll get conflicts on all paths where content differs.
How can one both get the file "coloring" and add own dependencies with a script to an rpm? Iirc the internal dependency generator needs to be disabled to be able to specify a special script to add dependencies.
The sorry state of things right now is that there's no generic way that works in every situation with the internal dependency generator, but for some cases its possible:
- All the require/provide helper scripts the internal dep generator uses can be overridden and customized, such as the __perl_provides/requires scripts.
The first step is probably to move all the actual dependency generation work out of librpmbuild into helpers that can be overridden and customized, so that 1) becomes a general (even if cumbersome) solution.
Is it maybe easily possible to just add another dummy helper script, that by default does not create any Requires/Provides, but is called for every RPM, e.g. __extra_provides/requires. Then this could be overwritten without disturbing the other dependency generation steps.
Regards Till
On Thu, Jan 28, 2010 at 11:21:02PM +0100, Till Maas wrote:
Is it maybe easily possible to just add another dummy helper script, that by default does not create any Requires/Provides, but is called for every RPM, e.g. __extra_provides/requires. Then this could be overwritten without disturbing the other dependency generation steps.
That wouldn't be sufficiently flexible. What you want is let the "internal dependency generator" (either as something in librpmbuild or a separate helper program) produce text output, let it filter through some optional script and parse/use the result. Then you can filter out some unwanted dependencies, or add them, or modify...
Jakub
-
Bill Nottingham -
Jakub Jelinek -
Panu Matilainen -
Richard W.M. Jones -
Till Maas