For those who aren't familiar, QEMU actually provides two completely different sets of emulators
- system emulators - they emulate a full virtual machine and thus run a full guest OS. - user emulators - they emulate the Linux userspace ABI letting you run non-native arch executables directly.
The user emulators are what I'm concerned with in this mail, so ignore the system emulators.
Currently all the user emulators are provided in the "qemu-user" RPM which also includes files in /usr/lib/binfmt.d to register each emulator binary as a binary format handler for its respective architecture.
This is ok if you have a non-native arch binary that's statically linked and you just want to run it from context of your main OS root filesystem. Running dynamic linked binaries won't fly because if say running an arm binary on x86_64 host, it'll look for /lib/libc.so and find the i386 one, instead of the arm one. You can't set LD_LIBRARY_PATH to override this as the env var will apply to both qemu-arm (an x86_64 binary) and the binary it is trying to run (an arm binary).
More typical though is that you have a directory containing an fullish install tree of a non-native architecture and you just want to chroot into that. When doing such a chroot, the qemu-$ARCH emulator must be present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm. So again you have the potential problem of clashing libc.so in /usr/lib It is a shame Fedora doesn't have full multi-arch support, instead of merely multi-lib to avoid these clashing lib dirs across architecture RPMs.
The recommended way to deal with this for the qemu user emulator binaries to be statically linked, so when copied inside the non-native arch chroot, they never need to resolve any native arch libraries. Fedora's qemu user binaries are all dynamic linked right now.
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries - qemu-binfmt - binfmt rules registering the dynamic linked binaries - qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
NB, this means qemu-binfmt and qemu-user-static are mutually exclusive since they both provide the same binfmt files. You can however have both qemu-user and qemu-user-static installed as their binary names won't clash, and in this case the static ones will be registered as binfmts
This nice thing about this multiple package approach is that when you copied the x86_64 build of the "qemu-arm-static" binary into your arm chroot, you still then have the possibility of installing the arm build of the "qemu-arm" binary inside that chroot without filename clash.
An alternative simpler approach would be to just have one package, qemu-user, which contains the static binaries and never ship any dynamic linked qemu user binaries. This is slightly more restrictive though, as explained in the previous paragraph, so I'd like to avoid doing that.
I'd like to make using non-native arch chroots simple with Fedora without people needing to manually build their own static QEMU binaries, or download static binaries provided by another distro[2]. So I'm suggesting to make a change to Fedora qemu packages to essentially copy the way Debian has done things. Specifically I will
- Pull the binfmt registration files out of qemu-user and into a new qemu-binfmt package which depends on qemu-user.
- Add static builds of qemu user emulators to a new qemu-user-static package, along with binfmt registration files
The static build of QEMU user emulators is moderately light on dependancies, only requiring glib2-static, pcre-static, zlib-static and glibc-static packages.
The change to introduce a qemu-binfmt package has small upgrade implications since anyone with qemu-user installed today, will loose the binary format rules unless they manually install qemu-binfmt. I think the number of people affected is probably quite small, and some of them may well wish to use qemu-user-static instead anyway.
Obviously this would only be done in rawhide, not any existing stable releases of Fedora.
Nothing will change about the rest of QEMU packaging - ie all system emulators will continue to use dynamic linking
Regards, Daniel
[1] https://wiki.debian.org/QemuUserEmulation [2] https://rwmj.wordpress.com/2013/12/22/how-to-run-aarch64-binaries-on-an-x86-...
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
Debian builds static libraries and puts them into their -dev packages. Fedora does not do this systematically. Are all the static libraries you need actually available in Fedora?
An alternative would be to bind-mount the real file system into the chroot and run the native dynamic linker with a --library-path command line argument that specifies the library directories in the bind mount. (It's reasonable to specify --inhibit-cache as well.) This would need some adjustments in the binfmt wrapper.
Florian
On Wed, Jun 29, 2016 at 12:15:02PM +0200, Florian Weimer wrote:
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
Debian builds static libraries and puts them into their -dev packages. Fedora does not do this systematically. Are all the static libraries you need actually available in Fedora?
Yes, everything we need exists - as mentioned its only glibc-static, glib2-static, zlib-static and pcre-static that we need for this.
An alternative would be to bind-mount the real file system into the chroot and run the native dynamic linker with a --library-path command line argument that specifies the library directories in the bind mount. (It's reasonable to specify --inhibit-cache as well.) This would need some adjustments in the binfmt wrapper.
The binfmt registrations are global to the OS, so the same binfmt command needs to work whether inside or outside the chroot. So a scheme which requires us to pass special args to make the linker looks elsewhere when inside the chroot is not so easy. We'd have to create a wrapper program around the real qemu user binary that decides which args to pass to the linker, and that wrapper would then have to be statically linked
Regards, Daniel
On Wed, 2016-06-29 at 11:34 +0100, Daniel P. Berrange wrote:
On Wed, Jun 29, 2016 at 12:15:02PM +0200, Florian Weimer wrote:
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
Debian builds static libraries and puts them into their -dev packages. Fedora does not do this systematically. Are all the static libraries you need actually available in Fedora?
Yes, everything we need exists - as mentioned its only glibc-static, glib2-static, zlib-static and pcre-static that we need for this.
An alternative would be to bind-mount the real file system into the chroot and run the native dynamic linker with a --library-path command line argument that specifies the library directories in the bind mount. (It's reasonable to specify --inhibit-cache as well.) This would need some adjustments in the binfmt wrapper.
The binfmt registrations are global to the OS, so the same binfmt command needs to work whether inside or outside the chroot. So a scheme which requires us to pass special args to make the linker looks elsewhere when inside the chroot is not so easy. We'd have to create a wrapper program around the real qemu user binary that decides which args to pass to the linker, and that wrapper would then have to be statically linked
Just an idea:
Would it make sense to build the qemu-user binary so that it looks in a different /lib directory (using rpath) like /lib/qemu-user-systemlib, and then symlink into that directory the libraries it needs from the host ?
That way if you use chroot and bind mount in that directory in the right place the qemu-user binary uses different libraries from the emulated binaries yet you do not have to resort to static linking.
Simo.
On Wed, Jun 29, 2016 at 09:39:11AM -0400, Simo Sorce wrote:
On Wed, 2016-06-29 at 11:34 +0100, Daniel P. Berrange wrote:
On Wed, Jun 29, 2016 at 12:15:02PM +0200, Florian Weimer wrote:
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
Debian builds static libraries and puts them into their -dev packages. Fedora does not do this systematically. Are all the static libraries you need actually available in Fedora?
Yes, everything we need exists - as mentioned its only glibc-static, glib2-static, zlib-static and pcre-static that we need for this.
An alternative would be to bind-mount the real file system into the chroot and run the native dynamic linker with a --library-path command line argument that specifies the library directories in the bind mount. (It's reasonable to specify --inhibit-cache as well.) This would need some adjustments in the binfmt wrapper.
The binfmt registrations are global to the OS, so the same binfmt command needs to work whether inside or outside the chroot. So a scheme which requires us to pass special args to make the linker looks elsewhere when inside the chroot is not so easy. We'd have to create a wrapper program around the real qemu user binary that decides which args to pass to the linker, and that wrapper would then have to be statically linked
Just an idea:
Would it make sense to build the qemu-user binary so that it looks in a different /lib directory (using rpath) like /lib/qemu-user-systemlib, and then symlink into that directory the libraries it needs from the host ?
That way if you use chroot and bind mount in that directory in the right place the qemu-user binary uses different libraries from the emulated binaries yet you do not have to resort to static linking.
I think you would need multiple levels of symlinking
On the host root
symlink: /usr/lib/qemu-user-root -> / dir: /usr/lib/qemu-user-host-lib symlink: /usr/lib/qemu-user-host-lib/libc.so -> /usr/lib/qemu-user-root/lib/libc.so
And build qemu-$ARCH with rpath to /usr/lib/qemu-user-root/lib.
Then in the chroot you would need to bind mount the host root into /usr/lib/qemu-user-root, to override the symlink that would otherwise point back to /
Even if you do all that, you've now prevented installation of the foreign arch's qemu-user RPM inside the chroot, as that will clash with the one you've setup from the host. Also the way you deal with cross-arch chroots is now different (and more complex) on Fedora, vs every other Linux distro which just provides a static qemu-$ARCH you can copy without needing to care about bind mounting extra dirs.
So yes, it is probably possible, but it is not very appealing IMHO
Regards, Daniel
On Wed, 2016-06-29 at 15:02 +0100, Daniel P. Berrange wrote:
On Wed, Jun 29, 2016 at 09:39:11AM -0400, Simo Sorce wrote:
On Wed, 2016-06-29 at 11:34 +0100, Daniel P. Berrange wrote:
On Wed, Jun 29, 2016 at 12:15:02PM +0200, Florian Weimer wrote:
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
Debian builds static libraries and puts them into their -dev packages. Fedora does not do this systematically. Are all the static libraries you need actually available in Fedora?
Yes, everything we need exists - as mentioned its only glibc-static, glib2-static, zlib-static and pcre-static that we need for this.
An alternative would be to bind-mount the real file system into the chroot and run the native dynamic linker with a --library-path command line argument that specifies the library directories in the bind mount. (It's reasonable to specify --inhibit-cache as well.) This would need some adjustments in the binfmt wrapper.
The binfmt registrations are global to the OS, so the same binfmt command needs to work whether inside or outside the chroot. So a scheme which requires us to pass special args to make the linker looks elsewhere when inside the chroot is not so easy. We'd have to create a wrapper program around the real qemu user binary that decides which args to pass to the linker, and that wrapper would then have to be statically linked
Just an idea:
Would it make sense to build the qemu-user binary so that it looks in a different /lib directory (using rpath) like /lib/qemu-user-systemlib, and then symlink into that directory the libraries it needs from the host ?
That way if you use chroot and bind mount in that directory in the right place the qemu-user binary uses different libraries from the emulated binaries yet you do not have to resort to static linking.
I think you would need multiple levels of symlinking
On the host root
symlink: /usr/lib/qemu-user-root -> / dir: /usr/lib/qemu-user-host-lib symlink: /usr/lib/qemu-user-host-lib/libc.so -> /usr/lib/qemu-user-root/lib/libc.so
And build qemu-$ARCH with rpath to /usr/lib/qemu-user-root/lib.
Then in the chroot you would need to bind mount the host root into /usr/lib/qemu-user-root, to override the symlink that would otherwise point back to /
Why do you need 2 symlinks ?
I was thinking you'd have symlink: /usr/lib/qemu-user-lib-x86_64/ -> /usr/lib
In the chroot you just bind mount the host's /usr/lib as /usr/lib/qemu-user-lib-x86_64
This assuming x86-64, you can bind mount in the appropriate arch-target dir on other arches.
Even if you do all that, you've now prevented installation of the foreign arch's qemu-user RPM inside the chroot,
See above, I think my solution would not prevent that.
as that will clash with the one you've setup from the host. Also the way you deal with cross-arch chroots is now different (and more complex) on Fedora, vs every other Linux distro which just provides a static qemu-$ARCH you can copy without needing to care about bind mounting extra dirs.
Yes, bind-mount is an additional step, but doesn't look like a huge price to pay. Could even be done transparently by providing a qemu-user.sh script that "does the right thing"
So yes, it is probably possible, but it is not very appealing IMHO
Thought it may be neat, but if consistency with other distros is what you are after then I would also lean to build a static binary. It's just that linking in glibc statically is ... not great.
Simo.
On 29/06/2016 16:27, Simo Sorce wrote:
symlink: /usr/lib/qemu-user-root -> / dir: /usr/lib/qemu-user-host-lib symlink: /usr/lib/qemu-user-host-lib/libc.so -> /usr/lib/qemu-user-root/lib/libc.so
And build qemu-$ARCH with rpath to /usr/lib/qemu-user-root/lib.
Then in the chroot you would need to bind mount the host root into /usr/lib/qemu-user-root, to override the symlink that would otherwise point back to /
Why do you need 2 symlinks ?
I was thinking you'd have symlink: /usr/lib/qemu-user-lib-x86_64/ -> /usr/lib
In the chroot you just bind mount the host's /usr/lib as /usr/lib/qemu-user-lib-x86_64
This assuming x86-64, you can bind mount in the appropriate arch-target dir on other arches.
This only works if all symlinks in /usr/lib are relative. Even then, there are some cases where /usr/lib/foo.so symlinks to ../../lib/foo.so.1, so you'd need to:
1) bind-mount /usr/lib into /usr/lib/qemu-user-root/usr/lib
2) create a symlink /usr/lib/qemu-user-root/lib to /usr/lib/qemu-user-root/usr/lib.
It seems a bit brittle.
On 06/29/2016 12:34 PM, Daniel P. Berrange wrote:
On Wed, Jun 29, 2016 at 12:15:02PM +0200, Florian Weimer wrote:
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
Debian builds static libraries and puts them into their -dev packages. Fedora does not do this systematically. Are all the static libraries you need actually available in Fedora?
Yes, everything we need exists - as mentioned its only glibc-static, glib2-static, zlib-static and pcre-static that we need for this.
In this case, linking statically is indeed an option.
We currently have little tooling for static libraries. Strictly speaking, all statically linked libraries have to be rebuilt even for minor glibc updates because we do not provide ABI compatibility for static linking (there are no compatibility symbols, static binaries always get the most recent implementation).
An alternative would be to bind-mount the real file system into the chroot and run the native dynamic linker with a --library-path command line argument that specifies the library directories in the bind mount. (It's reasonable to specify --inhibit-cache as well.) This would need some adjustments in the binfmt wrapper.
The binfmt registrations are global to the OS, so the same binfmt command needs to work whether inside or outside the chroot. So a scheme which requires us to pass special args to make the linker looks elsewhere when inside the chroot is not so easy. We'd have to create a wrapper program around the real qemu user binary that decides which args to pass to the linker, and that wrapper would then have to be statically linked
Yes, but it would be much simpler than the whole QEMU binary with its library dependencies.
Florian
On Wed, Jun 29, 2016 at 04:54:42PM +0200, Florian Weimer wrote:
On 06/29/2016 12:34 PM, Daniel P. Berrange wrote:
On Wed, Jun 29, 2016 at 12:15:02PM +0200, Florian Weimer wrote:
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
Debian builds static libraries and puts them into their -dev packages. Fedora does not do this systematically. Are all the static libraries you need actually available in Fedora?
Yes, everything we need exists - as mentioned its only glibc-static, glib2-static, zlib-static and pcre-static that we need for this.
In this case, linking statically is indeed an option.
We currently have little tooling for static libraries. Strictly speaking, all statically linked libraries have to be rebuilt even for minor glibc updates because we do not provide ABI compatibility for static linking (there are no compatibility symbols, static binaries always get the most recent implementation).
Yep, I understand those caveats - the flipside is that most other Linux distros already provide statically linked qemu user emulators with glibc without suffering unduly. So while I accept there are potential problems, it doesn't seem like they're frequent / severe enough to make this approach unviable.
Regards, Daniel
On Wed, Jun 29, 2016 at 3:03 AM, Daniel P. Berrange berrange@redhat.com wrote:
For those who aren't familiar, QEMU actually provides two completely different sets of emulators
- system emulators - they emulate a full virtual machine and thus run a full guest OS.
- user emulators - they emulate the Linux userspace ABI letting you run non-native arch executables directly.
The user emulators are what I'm concerned with in this mail, so ignore the system emulators.
Currently all the user emulators are provided in the "qemu-user" RPM which also includes files in /usr/lib/binfmt.d to register each emulator binary as a binary format handler for its respective architecture.
This is ok if you have a non-native arch binary that's statically linked and you just want to run it from context of your main OS root filesystem. Running dynamic linked binaries won't fly because if say running an arm binary on x86_64 host, it'll look for /lib/libc.so and find the i386 one, instead of the arm one. You can't set LD_LIBRARY_PATH to override this as the env var will apply to both qemu-arm (an x86_64 binary) and the binary it is trying to run (an arm binary).
More typical though is that you have a directory containing an fullish install tree of a non-native architecture and you just want to chroot into that. When doing such a chroot, the qemu-$ARCH emulator must be present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm. So again you have the potential problem of clashing libc.so in /usr/lib It is a shame Fedora doesn't have full multi-arch support, instead of merely multi-lib to avoid these clashing lib dirs across architecture RPMs.
The recommended way to deal with this for the qemu user emulator binaries to be statically linked, so when copied inside the non-native arch chroot, they never need to resolve any native arch libraries. Fedora's qemu user binaries are all dynamic linked right now.
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
NB, this means qemu-binfmt and qemu-user-static are mutually exclusive since they both provide the same binfmt files. You can however have both qemu-user and qemu-user-static installed as their binary names won't clash, and in this case the static ones will be registered as binfmts
This nice thing about this multiple package approach is that when you copied the x86_64 build of the "qemu-arm-static" binary into your arm chroot, you still then have the possibility of installing the arm build of the "qemu-arm" binary inside that chroot without filename clash.
An alternative simpler approach would be to just have one package, qemu-user, which contains the static binaries and never ship any dynamic linked qemu user binaries. This is slightly more restrictive though, as explained in the previous paragraph, so I'd like to avoid doing that.
I'd like to make using non-native arch chroots simple with Fedora without people needing to manually build their own static QEMU binaries, or download static binaries provided by another distro[2]. So I'm suggesting to make a change to Fedora qemu packages to essentially copy the way Debian has done things. Specifically I will
Pull the binfmt registration files out of qemu-user and into a new qemu-binfmt package which depends on qemu-user.
Add static builds of qemu user emulators to a new qemu-user-static package, along with binfmt registration files
The static build of QEMU user emulators is moderately light on dependancies, only requiring glib2-static, pcre-static, zlib-static and glibc-static packages.
The change to introduce a qemu-binfmt package has small upgrade implications since anyone with qemu-user installed today, will loose the binary format rules unless they manually install qemu-binfmt. I think the number of people affected is probably quite small, and some of them may well wish to use qemu-user-static instead anyway.
Obviously this would only be done in rawhide, not any existing stable releases of Fedora.
Nothing will change about the rest of QEMU packaging - ie all system emulators will continue to use dynamic linking
Regards, Daniel
Amazingly, I've been pre-empted here. I was going to ask about this for Fedora, since we've got interest in this for Mageia as part of potentially enabling cross-arch building with mock in an easy fashion.
Mageia tries to keep its QEMU package in sync with Fedora, so I'd love to see this happen so that it becomes easier to maintain the qemu-user-static stuff.
Right now, we only have a couple of the arches built, but one of our guys interested in this might be willing to help out on this side of the fence with it (Per Øyvind Karlsen). I've CC'd him to this email.
On Wed, Jun 29, 2016 at 06:45:36AM -0700, Neal Gompa wrote:
On Wed, Jun 29, 2016 at 3:03 AM, Daniel P. Berrange berrange@redhat.com wrote:
For those who aren't familiar, QEMU actually provides two completely different sets of emulators
- system emulators - they emulate a full virtual machine and thus run a full guest OS.
- user emulators - they emulate the Linux userspace ABI letting you run non-native arch executables directly.
The user emulators are what I'm concerned with in this mail, so ignore the system emulators.
Currently all the user emulators are provided in the "qemu-user" RPM which also includes files in /usr/lib/binfmt.d to register each emulator binary as a binary format handler for its respective architecture.
This is ok if you have a non-native arch binary that's statically linked and you just want to run it from context of your main OS root filesystem. Running dynamic linked binaries won't fly because if say running an arm binary on x86_64 host, it'll look for /lib/libc.so and find the i386 one, instead of the arm one. You can't set LD_LIBRARY_PATH to override this as the env var will apply to both qemu-arm (an x86_64 binary) and the binary it is trying to run (an arm binary).
More typical though is that you have a directory containing an fullish install tree of a non-native architecture and you just want to chroot into that. When doing such a chroot, the qemu-$ARCH emulator must be present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm. So again you have the potential problem of clashing libc.so in /usr/lib It is a shame Fedora doesn't have full multi-arch support, instead of merely multi-lib to avoid these clashing lib dirs across architecture RPMs.
The recommended way to deal with this for the qemu user emulator binaries to be statically linked, so when copied inside the non-native arch chroot, they never need to resolve any native arch libraries. Fedora's qemu user binaries are all dynamic linked right now.
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
NB, this means qemu-binfmt and qemu-user-static are mutually exclusive since they both provide the same binfmt files. You can however have both qemu-user and qemu-user-static installed as their binary names won't clash, and in this case the static ones will be registered as binfmts
This nice thing about this multiple package approach is that when you copied the x86_64 build of the "qemu-arm-static" binary into your arm chroot, you still then have the possibility of installing the arm build of the "qemu-arm" binary inside that chroot without filename clash.
An alternative simpler approach would be to just have one package, qemu-user, which contains the static binaries and never ship any dynamic linked qemu user binaries. This is slightly more restrictive though, as explained in the previous paragraph, so I'd like to avoid doing that.
I'd like to make using non-native arch chroots simple with Fedora without people needing to manually build their own static QEMU binaries, or download static binaries provided by another distro[2]. So I'm suggesting to make a change to Fedora qemu packages to essentially copy the way Debian has done things. Specifically I will
Pull the binfmt registration files out of qemu-user and into a new qemu-binfmt package which depends on qemu-user.
Add static builds of qemu user emulators to a new qemu-user-static package, along with binfmt registration files
The static build of QEMU user emulators is moderately light on dependancies, only requiring glib2-static, pcre-static, zlib-static and glibc-static packages.
The change to introduce a qemu-binfmt package has small upgrade implications since anyone with qemu-user installed today, will loose the binary format rules unless they manually install qemu-binfmt. I think the number of people affected is probably quite small, and some of them may well wish to use qemu-user-static instead anyway.
Obviously this would only be done in rawhide, not any existing stable releases of Fedora.
Nothing will change about the rest of QEMU packaging - ie all system emulators will continue to use dynamic linking
Regards, Daniel
Amazingly, I've been pre-empted here. I was going to ask about this for Fedora, since we've got interest in this for Mageia as part of potentially enabling cross-arch building with mock in an easy fashion.
Mageia tries to keep its QEMU package in sync with Fedora, so I'd love to see this happen so that it becomes easier to maintain the qemu-user-static stuff.
Right now, we only have a couple of the arches built, but one of our guys interested in this might be willing to help out on this side of the fence with it (Per Øyvind Karlsen). I've CC'd him to this email.
FYI, I have a test scratch-build with the proposed change:
http://koji.fedoraproject.org/koji/taskinfo?taskID=14702004
This is completely untested by me so far, so may have bugs,m but it serves to illustrate the goal.
Regards, Daniel
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
The change to introduce a qemu-binfmt package has small upgrade implications since anyone with qemu-user installed today, will loose the binary format rules unless they manually install qemu-binfmt.
Not if you add "Obsoletes: qemu-user < V-R" (where V-R is the first version-release that contains this change) to both qemu-binfmt and qemu-user.
Michal
On Thu, Jun 30, 2016 at 10:21:18AM +0200, Michal Schmidt wrote:
On 06/29/2016 12:03 PM, Daniel P. Berrange wrote:
The change to introduce a qemu-binfmt package has small upgrade implications since anyone with qemu-user installed today, will loose the binary format rules unless they manually install qemu-binfmt.
Not if you add "Obsoletes: qemu-user < V-R" (where V-R is the first version-release that contains this change) to both qemu-binfmt and qemu-user.
Oh, nice tip, thanks.
Regards, Daniel
"Daniel P. Berrange" berrange@redhat.com writes:
More typical though is that you have a directory containing an fullish install tree of a non-native architecture and you just want to chroot into that. When doing such a chroot, the qemu-$ARCH emulator must be present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm.
Hi, Dan,
Is this work from James Bottomley at all relevant?
http://comments.gmane.org/gmane.linux.file-systems/105033 http://blog.hansenpartnership.com/constructing-architecture-emulation-contai...
Cheers, Jeff
On Thu, Jul 07, 2016 at 09:48:24AM -0400, Jeff Moyer wrote:
"Daniel P. Berrange" berrange@redhat.com writes:
More typical though is that you have a directory containing an fullish install tree of a non-native architecture and you just want to chroot into that. When doing such a chroot, the qemu-$ARCH emulator must be present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm.
Hi, Dan,
Is this work from James Bottomley at all relevant?
http://comments.gmane.org/gmane.linux.file-systems/105033 http://blog.hansenpartnership.com/constructing-architecture-emulation-contai...
He's describing exactly the kind of approach that is common on other distros and that I want to work on Fedora too. His instructions are assuming use of a statically linked qem-$ARCH emulator too.
Regards, Daniel
On Wed, Jun 29, 2016 at 6:03 AM, Daniel P. Berrange berrange@redhat.com wrote:
For those who aren't familiar, QEMU actually provides two completely different sets of emulators
- system emulators - they emulate a full virtual machine and thus run a full guest OS.
- user emulators - they emulate the Linux userspace ABI letting you run non-native arch executables directly.
The user emulators are what I'm concerned with in this mail, so ignore the system emulators.
Currently all the user emulators are provided in the "qemu-user" RPM which also includes files in /usr/lib/binfmt.d to register each emulator binary as a binary format handler for its respective architecture.
This is ok if you have a non-native arch binary that's statically linked and you just want to run it from context of your main OS root filesystem. Running dynamic linked binaries won't fly because if say running an arm binary on x86_64 host, it'll look for /lib/libc.so and find the i386 one, instead of the arm one. You can't set LD_LIBRARY_PATH to override this as the env var will apply to both qemu-arm (an x86_64 binary) and the binary it is trying to run (an arm binary).
More typical though is that you have a directory containing an fullish install tree of a non-native architecture and you just want to chroot into that. When doing such a chroot, the qemu-$ARCH emulator must be present inside the chroot too. ie the x86_64 build of /usr/bin/qemu-arm must be present inside at /my/chroot/for/fedora-arm/usr/bin/qemu-arm. So again you have the potential problem of clashing libc.so in /usr/lib It is a shame Fedora doesn't have full multi-arch support, instead of merely multi-lib to avoid these clashing lib dirs across architecture RPMs.
The recommended way to deal with this for the qemu user emulator binaries to be statically linked, so when copied inside the non-native arch chroot, they never need to resolve any native arch libraries. Fedora's qemu user binaries are all dynamic linked right now.
jfyi, nothing really to contribute on how to do this, but this has been a long-standing gripe of mine about fedora, and somehow packaging qemu-static is something I'm a really big fan of.
(in a previous life, using sbox2 to "natively" build arm stuff on x86 was quite useful..)
BR, -R
Debian handles this by having several packages [1]
- qemu-user - the dynamic linked qemu user binaries
- qemu-binfmt - binfmt rules registering the dynamic linked binaries
- qemu-user-static - the static linked qemu user binaries *and* binfmt rules to register them. The static binaries all have -static suffix on their name
NB, this means qemu-binfmt and qemu-user-static are mutually exclusive since they both provide the same binfmt files. You can however have both qemu-user and qemu-user-static installed as their binary names won't clash, and in this case the static ones will be registered as binfmts
This nice thing about this multiple package approach is that when you copied the x86_64 build of the "qemu-arm-static" binary into your arm chroot, you still then have the possibility of installing the arm build of the "qemu-arm" binary inside that chroot without filename clash.
An alternative simpler approach would be to just have one package, qemu-user, which contains the static binaries and never ship any dynamic linked qemu user binaries. This is slightly more restrictive though, as explained in the previous paragraph, so I'd like to avoid doing that.
I'd like to make using non-native arch chroots simple with Fedora without people needing to manually build their own static QEMU binaries, or download static binaries provided by another distro[2]. So I'm suggesting to make a change to Fedora qemu packages to essentially copy the way Debian has done things. Specifically I will
Pull the binfmt registration files out of qemu-user and into a new qemu-binfmt package which depends on qemu-user.
Add static builds of qemu user emulators to a new qemu-user-static package, along with binfmt registration files
The static build of QEMU user emulators is moderately light on dependancies, only requiring glib2-static, pcre-static, zlib-static and glibc-static packages.
The change to introduce a qemu-binfmt package has small upgrade implications since anyone with qemu-user installed today, will loose the binary format rules unless they manually install qemu-binfmt. I think the number of people affected is probably quite small, and some of them may well wish to use qemu-user-static instead anyway.
Obviously this would only be done in rawhide, not any existing stable releases of Fedora.
Nothing will change about the rest of QEMU packaging - ie all system emulators will continue to use dynamic linking
Regards, Daniel
[1] https://wiki.debian.org/QemuUserEmulation [2] https://rwmj.wordpress.com/2013/12/22/how-to-run-aarch64-binaries-on-an-x86-... -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- devel mailing list devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
Daniel, thanks for presenting this so clearly; I think you're spot-on. Has there been any progress on this?
On Sat, Jan 14, 2017 at 11:13 AM, Jonathon Reinhart jonathon.reinhart@gmail.com wrote:
Daniel, thanks for presenting this so clearly; I think you're spot-on. Has there been any progress on this?
qemu-user-static has been shipping in Fedora qemu builds since qemu 2.6.0-5 in Fedora 24. :)
-
Daniel P. Berrange -
Florian Weimer -
Jeff Moyer -
Jonathon Reinhart -
Michal Schmidt -
Neal Gompa -
Paolo Bonzini -
Rob Clark -
Simo Sorce