(This message was originally sent to the Packaging mailing list, where Jason Tibbitts pointed that this is a restriction requested by the Kernel team, and it'll be your opinion the one that will prevail here)
Hi,
The document "What can be packaged" from "Fedora Packaging Guidelines", in the section "Only one kernel package" [1], states that "Fedora allows only a single kernel package; packages containing alternate kernels are not allowed in the distribution."
While not explicitly stated there, I suspect (please correct me if I'm wrong) that statement was written with the idea of preventing alternate kernels that could be used to boot the system. With this premise in mind, I was wondering if non-bootable kernels (that is, kernels in a binary format that's not accepted by a conventional boot loader) would be accepted for packaging.
I'm asking this because I would like to package "libkrunfw" [2], a dynamic library that bundles an slightly modified minimalist Linux kernel. The library doesn't really link against the kernel (in the sense that it doesn't resolve any symbols nor calls to any of its code), it just bundles it in a binary format that allows it to be directly injected in a KVM memory region, so it's quite similar to a compressed image format, but for a different use case.
"libkrunfw" is consumed by "libkrun" [3], another dynamic library that allows programs to acquire virtualization-based process isolation capabilites. The main user of "libkrun" is "crun", when built with "--with-libkrun", an OCI runtime used by "podman". When all pieces are in place, users can easily run containers with virtualization-based isolation by adding some additional flags to the "podman" command line. I have a COPR repository with pre-built alternative packages as a demonstration [4].
There are a number of reasons why we can't use the kernel that ships with Fedora:
- We carry a small number of patches with minor changes that modify the behavior of the kernel for this particular use case. Without them, we can't provide an streamlined UX for running isolated processes.
- We need an aggressive minimalist configuration to reduce the memory footprint of each container/isolated process.
- We need it to be bundled in a dynamic library, so their contents are mapped into the process memory, enabling programs to switch between namespaces without the need to carry the kernel binary with them. The binary object also needs to be properly aligned to allow direct injection into the KVM memory region without additional copies.
Given that "libkrunfw" bundles a kernel image that can't be used for booting the system, would it be acceptable to package it in Fedora?
Thanks, Sergio.
[1] https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-packag... [2] https://github.com/containers/libkrunfw [3] https://github.com/containers/libkrun [4] https://copr.fedorainfracloud.org/coprs/slp/crun-krun/
On Thu, Nov 12, 2020 at 2:38 AM Sergio Lopez slp@redhat.com wrote:
(This message was originally sent to the Packaging mailing list, where Jason Tibbitts pointed that this is a restriction requested by the Kernel team, and it'll be your opinion the one that will prevail here)
Hi,
The document "What can be packaged" from "Fedora Packaging Guidelines", in the section "Only one kernel package" [1], states that "Fedora allows only a single kernel package; packages containing alternate kernels are not allowed in the distribution."
While not explicitly stated there, I suspect (please correct me if I'm wrong) that statement was written with the idea of preventing alternate kernels that could be used to boot the system. With this premise in mind, I was wondering if non-bootable kernels (that is, kernels in a binary format that's not accepted by a conventional boot loader) would be accepted for packaging.
I'm asking this because I would like to package "libkrunfw" [2], a dynamic library that bundles an slightly modified minimalist Linux kernel. The library doesn't really link against the kernel (in the sense that it doesn't resolve any symbols nor calls to any of its code), it just bundles it in a binary format that allows it to be directly injected in a KVM memory region, so it's quite similar to a compressed image format, but for a different use case.
"libkrunfw" is consumed by "libkrun" [3], another dynamic library that allows programs to acquire virtualization-based process isolation capabilites. The main user of "libkrun" is "crun", when built with "--with-libkrun", an OCI runtime used by "podman". When all pieces are in place, users can easily run containers with virtualization-based isolation by adding some additional flags to the "podman" command line. I have a COPR repository with pre-built alternative packages as a demonstration [4].
There are a number of reasons why we can't use the kernel that ships with Fedora:
- We carry a small number of patches with minor changes that modify the behavior of the kernel for this particular use case. Without them, we can't provide an streamlined UX for running isolated processes.
Are these patches not headed upstream? How are these handled with regards to security updates for the kernel itself, how current do these patches run? The Fedora kernel tends to be updated once to twice a week for released versions of Fedora, and we typically have a number of CVEs flowing in. With this smaller kernel, some of those will not be relevant, but someone would need to keep up with them, determine if they are or are not, and apply them if so. Of course it might just be easier to just always build.
- We need an aggressive minimalist configuration to reduce the memory footprint of each container/isolated process.
There could be other ways to accomplish this.
- We need it to be bundled in a dynamic library, so their contents are mapped into the process memory, enabling programs to switch between namespaces without the need to carry the kernel binary with them. The binary object also needs to be properly aligned to allow direct injection into the KVM memory region without additional copies.
Given that "libkrunfw" bundles a kernel image that can't be used for booting the system, would it be acceptable to package it in Fedora?
It does alleviate some of the concerns, there wouldn't be bugs filed against the kernel package where we have to figure out what kernel is being booted, etc. I am still very much concerned about how updates/security are handled in such a scenario though.
Justin
On Thu, Nov 12, 2020 at 01:18:21PM -0600, Justin Forbes wrote:
On Thu, Nov 12, 2020 at 2:38 AM Sergio Lopez slp@redhat.com wrote:
(This message was originally sent to the Packaging mailing list, where Jason Tibbitts pointed that this is a restriction requested by the Kernel team, and it'll be your opinion the one that will prevail here)
Hi,
The document "What can be packaged" from "Fedora Packaging Guidelines", in the section "Only one kernel package" [1], states that "Fedora allows only a single kernel package; packages containing alternate kernels are not allowed in the distribution."
While not explicitly stated there, I suspect (please correct me if I'm wrong) that statement was written with the idea of preventing alternate kernels that could be used to boot the system. With this premise in mind, I was wondering if non-bootable kernels (that is, kernels in a binary format that's not accepted by a conventional boot loader) would be accepted for packaging.
I'm asking this because I would like to package "libkrunfw" [2], a dynamic library that bundles an slightly modified minimalist Linux kernel. The library doesn't really link against the kernel (in the sense that it doesn't resolve any symbols nor calls to any of its code), it just bundles it in a binary format that allows it to be directly injected in a KVM memory region, so it's quite similar to a compressed image format, but for a different use case.
"libkrunfw" is consumed by "libkrun" [3], another dynamic library that allows programs to acquire virtualization-based process isolation capabilites. The main user of "libkrun" is "crun", when built with "--with-libkrun", an OCI runtime used by "podman". When all pieces are in place, users can easily run containers with virtualization-based isolation by adding some additional flags to the "podman" command line. I have a COPR repository with pre-built alternative packages as a demonstration [4].
There are a number of reasons why we can't use the kernel that ships with Fedora:
- We carry a small number of patches with minor changes that modify the behavior of the kernel for this particular use case. Without them, we can't provide an streamlined UX for running isolated processes.
Are these patches not headed upstream? How are these handled with regards to security updates for the kernel itself, how current do these patches run? The Fedora kernel tends to be updated once to twice a week for released versions of Fedora, and we typically have a number of CVEs flowing in. With this smaller kernel, some of those will not be relevant, but someone would need to keep up with them, determine if they are or are not, and apply them if so. Of course it might just be easier to just always build.
The intention is to keep the gap as small as possible by trying to upstream all changes. Probably there will be a some of them that won't make upstream, but those will be super-small changes (such as not complaining loudly if init gets killed) that only make sense for this particular use case.
The patches are based on 5.8, but will be rebased on 5.10 as soon its final version is released.
- We need an aggressive minimalist configuration to reduce the memory footprint of each container/isolated process.
There could be other ways to accomplish this.
Our intention is too keep the memory footprint under 50 MiB in total. That's accounting for the weight of the not shared regions of the dynamic libraries, the r/w sections of the kernel, and the memory used by the kernel inside the VM.
To achieve that, we rely on very aggressive optimizations such as disabling modularity, enabling options for embedded systems, reducing the maximum number of cores and memory, disabling most options and drivers, optimizing for size (-Os)...
- We need it to be bundled in a dynamic library, so their contents are mapped into the process memory, enabling programs to switch between namespaces without the need to carry the kernel binary with them. The binary object also needs to be properly aligned to allow direct injection into the KVM memory region without additional copies.
Given that "libkrunfw" bundles a kernel image that can't be used for booting the system, would it be acceptable to package it in Fedora?
It does alleviate some of the concerns, there wouldn't be bugs filed against the kernel package where we have to figure out what kernel is being booted, etc. I am still very much concerned about how updates/security are handled in such a scenario though.
I think security shouldn't be a concern here. This may sound silly, but there's a good explanation for it. This kernel will be used by libkrun (which integrates Virtual Machine Monitor functionality) to operate small Virtual Machines where processes that would normally run directly on the host, would run isolated within the VM.
The VMM will operate in the same security context as the original process. That is, if the process is the entry point of a container created by podman, the VMM will run in the same namespaces and security profile that the original containerized process would have run.
In this scenario, from the host's perspective, even if the guest's kernel suffers some kind of vulnerability, it would still be more secure than running the container without the VM, as the attack surface is smaller (it's the VMM instead arbitrary binaries inside a container), it'll have a more precise seccomp profile (we know the syscalls used by the VMM) and it's still running in the container's security context.
In other words, a CVE in the kernel bundled in libkrunfw won't have any impact on the host's security.
Thanks, Sergio.
On Thu, 12 Nov 2020 21:56:55 +0100, Sergio Lopez wrote:
The intention is to keep the gap as small as possible by trying to upstream all changes. Probably there will be a some of them that won't make upstream, but those will be super-small changes (such as not complaining loudly if init gets killed) that only make sense for this particular use case.
All patches should be upstream. There's no reason for keeping anything non-upstream in your project.
The patches are based on 5.8, but will be rebased on 5.10 as soon its final version is released.
The fact that you're playing a catch up game and don't even have 5.9 does not bring confidence that security problems can be quickly addressed.
To achieve that, we rely on very aggressive optimizations such as disabling modularity, enabling options for embedded systems, reducing the maximum number of cores and memory, disabling most options and drivers, optimizing for size (-Os)...
What you're describing is just a different kernel config. Such kernel should be built from Fedora kernel sources using a different kernel config.
In other words, a CVE in the kernel bundled in libkrunfw won't have any impact on the host's security.
Even if this was the case, it would still be a security issue in the application and as such should be promptly fixed.
Jiri
On Fri, Nov 13, 2020 at 10:09:04AM +0100, Jiri Benc wrote:
On Thu, 12 Nov 2020 21:56:55 +0100, Sergio Lopez wrote:
The intention is to keep the gap as small as possible by trying to upstream all changes. Probably there will be a some of them that won't make upstream, but those will be super-small changes (such as not complaining loudly if init gets killed) that only make sense for this particular use case.
All patches should be upstream. There's no reason for keeping anything non-upstream in your project.
Let me clear here, there's absolutely no intention of keeping things downstream, but there may be changes that won't be accepted upstream because they only make sense for this particular use case.
The patches are based on 5.8, but will be rebased on 5.10 as soon its final version is released.
The fact that you're playing a catch up game and don't even have 5.9 does not bring confidence that security problems can be quickly addressed.
We're not "playing a catch up game". Porting the changes to 5.10 [1] took less than one hour, because the changes are small and reasonably isolated (it'll be even more isolated once we reimplement the Transparent Socket Impersonation part as an independent address family).
The only reason we don't have a version based on 5.9 is because we didn't had the need for it. We want 5.10 because it incorporates the patch series for DAX support in virtio-fs, and also because it's LTS.
To achieve that, we rely on very aggressive optimizations such as disabling modularity, enabling options for embedded systems, reducing the maximum number of cores and memory, disabling most options and drivers, optimizing for size (-Os)...
What you're describing is just a different kernel config. Such kernel should be built from Fedora kernel sources using a different kernel config.
As I wrote in the original email, in addition of the downstream patches, the kernel is bundled in a different binary format so it can be directly injected into a KVM memory region.
In other words, a CVE in the kernel bundled in libkrunfw won't have any impact on the host's security.
Even if this was the case, it would still be a security issue in the application and as such should be promptly fixed.
No. Security issues in the VMM (Virtual Machine Monitor) or other functionality implemented in "libkrun" (which is a separate library and package than "libkrunfw", the one bundling the kernel) may have an impact on the host and must (and will be) promptly fixed.
Security issues in the bundled kernel can't have an impact on the host and it's _very_ unlikely the would have it on isolated application, as is operating within the security boundary of the VMM/VM, and the latter under the security context of the original application (in fact, a stricter context with a better seccomp profile).
This doesn't mean we can nor will be careless with what's bundled in libkrunfw, but it's definitely a few orders of magnitude less critical from a security perspective than a common, bootable kernel.
Thanks, Sergio.
On Fri, 13 Nov 2020 10:50:53 +0100, Sergio Lopez wrote:
Let me clear here, there's absolutely no intention of keeping things downstream, but there may be changes that won't be accepted upstream because they only make sense for this particular use case.
If they are not accepted upstream, it's for a reason and you need to address the reason. I very much doubt the reason for non-acceptance would be "we don't care about your use case". It may be "please make it more generic, so others will profit from it, too". Or "please do all of this differently". Which you should do instead of keeping your own patches.
There's no such thing as "we're special, thus we need own kernel patches". You're not special and you don't need them. Even Android finally got it.
We're not "playing a catch up game". Porting the changes to 5.10 [1] took less than one hour, because the changes are small and reasonably isolated (it'll be even more isolated once we reimplement the Transparent Socket Impersonation part as an independent address family).
"Transparent Socket Impersonation"? Oh. I don't see any such patches on the netdev mailing list. You'll have to start there, as it's very well possible you'll have some redesign to do. It's hard to get a kernel network feature right on the first attempt, the network stack is very focused on performance and maintainability.
For your next project, I suggest to start working with upstream communities from the very beginning. Developing whole infrastructure behind closed doors and then throwing the result at the communities is not a viable approach. Many have tried that before you, only to find out that upstreaming such solution takes more time than it took to develop it, rewriting pretty much everything along the way.
My advice to you is instead of "reimplement[ing] the Transparent Socket Impersonation part as an independent address family", reach out to netdev right away. Explain what you're trying to do and add a pointer to the current patches you have (or send them out as RFC directly if they're reasonably small). It will save you tons of time in the end.
As I wrote in the original email, in addition of the downstream patches, the kernel is bundled in a different binary format so it can be directly injected into a KVM memory region.
That's something that you'll need to first solve upstream in the kernel and KVM communities.
Security issues in the bundled kernel can't have an impact on the host and it's _very_ unlikely the would have it on isolated application, as is operating within the security boundary of the VMM/VM, and the latter under the security context of the original application (in fact, a stricter context with a better seccomp profile).
I don't think that downplaying security issues is a good approach.
Jiri
On Fri, Nov 13, 2020 at 11:33:21AM +0100, Jiri Benc wrote:
On Fri, 13 Nov 2020 10:50:53 +0100, Sergio Lopez wrote:
Let me clear here, there's absolutely no intention of keeping things downstream, but there may be changes that won't be accepted upstream because they only make sense for this particular use case.
If they are not accepted upstream, it's for a reason and you need to address the reason. I very much doubt the reason for non-acceptance would be "we don't care about your use case". It may be "please make it more generic, so others will profit from it, too". Or "please do all of this differently". Which you should do instead of keeping your own patches.
There's no such thing as "we're special, thus we need own kernel patches". You're not special and you don't need them. Even Android finally got it.
We're not "playing a catch up game". Porting the changes to 5.10 [1] took less than one hour, because the changes are small and reasonably isolated (it'll be even more isolated once we reimplement the Transparent Socket Impersonation part as an independent address family).
"Transparent Socket Impersonation"? Oh. I don't see any such patches on the netdev mailing list. You'll have to start there, as it's very well possible you'll have some redesign to do. It's hard to get a kernel network feature right on the first attempt, the network stack is very focused on performance and maintainability.
For your next project, I suggest to start working with upstream communities from the very beginning. Developing whole infrastructure behind closed doors and then throwing the result at the communities is not a viable approach. Many have tried that before you, only to find out that upstreaming such solution takes more time than it took to develop it, rewriting pretty much everything along the way.
My advice to you is instead of "reimplement[ing] the Transparent Socket Impersonation part as an independent address family", reach out to netdev right away. Explain what you're trying to do and add a pointer to the current patches you have (or send them out as RFC directly if they're reasonably small). It will save you tons of time in the end.
As I wrote in the original email, in addition of the downstream patches, the kernel is bundled in a different binary format so it can be directly injected into a KVM memory region.
That's something that you'll need to first solve upstream in the kernel and KVM communities.
Security issues in the bundled kernel can't have an impact on the host and it's _very_ unlikely the would have it on isolated application, as is operating within the security boundary of the VMM/VM, and the latter under the security context of the original application (in fact, a stricter context with a better seccomp profile).
I don't think that downplaying security issues is a good approach.
Jiri
Thanks for all the (unsolicited) advice. Deeply appreciated.
If possible, I would like to go back to my original question, so we can both go back to more fruitful matters.
Is it acceptable to package non-bootable kernels in Fedora?
Thanks, Sergio.
On Fri, 13 Nov 2020 11:55:17 +0100, Sergio Lopez wrote:
On Fri, Nov 13, 2020 at 11:51:47AM +0100, Jiri Benc wrote:
On Fri, 13 Nov 2020 11:42:32 +0100, Sergio Lopez wrote:
Is it acceptable to package non-bootable kernels in Fedora?
Based on what you described, I'd say no?
OK. Thanks for your prompt response.
You should probably wait for opinion of others on this list. I'm not the only one here and I don't get to decide. Especially Justin's opinion is more important than mine.
Jiri
Hey all. This is a Fedora project mailing list, and as such, we expect discourse to be friendly at all times, even when we disagree. Please remember to be excellent to each other. Thank you.
Hi,
Now that libkrun, libkrunfw and krunvm have landed in openSUSE Tumbleweed [1], I think it's a good time to reopen this discussion. I also had plenty of time to think about it, and my thoughts are now clearer, so please let me explain again why bundling a custom kernel with patches is a critical aspect of the libkrun project, and why this doesn't pose a risk for neither the Fedora Kernel Team, nor Fedora as a whole.
I like to define libkrun as "a dynamic library that enables other programs to easily gain Virtualization-based isolation capabilities, with the minimum possible footprint". While it does integrate the functionality of a VMM (Virtual Machine Monitor), in many aspects is closer to gVisor [2] than to QEMU [3], with the difference that gVisor uses a microkernel built for that specific purpose, while libkrun relies on a custom Linux kernel bundled in libkrunfw.
It's in the very nature of the project the need to explore building new bridges that allow the isolated workload running in the VM to communicate with the Host, in ways that are more appropriate to fulfill the needs of use cases such as "Lightweight VMs", "Virtualization-isolated Containers" or "Trusted Execution Environments (TEEs)". And this, out of necessity, implies extending both libkrun and the kernel bundled with it.
Now, I do understand why the actual kernel shipped with Fedora, the one that will be used by booting thousands of machines and will be running in privileged mode, needs to be held to sky-high standards, which may also include the restriction to not include additional patches, unless it's strongly justified. But, I also think that the kernel bundled in libkrunfw shouldn't be held by such standards.
The kernel bundled in libkrunfw can and only will be used by libkrun to bring up the Virtualized environment where the isolated workload will run. It will only be used in a Virtualization-isolated context, usually by unprivileged users, and in most cases in a pre-containerized environment. I argue, and this is a hill I'm willing to die on, that a bug in either the kernel bundled in libkrunfw or in libkrun itself, can't cause any more damage than any other userspace application shipped in Fedora. In fact, the risk is usually lower, given the context described before.
Having librunfw following the same high standards as the main kernel, we could end up going against our ability to provide new features to the users in a timely manner and gather a much needed feedback that's critical to the success of this project. In practice, this inhibits the possibility of using Fedora as the main upstream distribution for libkrun and the projects depending on it. And this would be very unfortunate, given the roots of libkrun
So, for the reasons stated above, I'd would like to reopen this question to see if together we can find a compromise that would work for all of us.
Thanks, Sergio.
[1] https://build.opensuse.org/package/show/openSUSE:Factory/libkrun [2] https://gvisor.dev/ [3] https://www.qemu.org/
Hi,
Now that libkrun, libkrunfw and krunvm have landed in openSUSE Tumbleweed [1], I think it's a good time to reopen this discussion. I also had plenty of time to think about it, and my thoughts are now clearer, so please let me explain again why bundling a custom kernel with patches is a critical aspect of the libkrun project, and why this doesn't pose a risk for neither the Fedora Kernel Team, nor Fedora as a whole.
I like to define libkrun as "a dynamic library that enables other programs to easily gain Virtualization-based isolation capabilities, with the minimum possible footprint". While it does integrate the functionality of a VMM (Virtual Machine Monitor), in many aspects is closer to gVisor [2] than to QEMU [3], with the difference that gVisor uses a microkernel built for that specific purpose, while libkrun relies on a custom Linux kernel bundled in libkrunfw.
It's in the very nature of the project the need to explore building new bridges that allow the isolated workload running in the VM to communicate with the Host, in ways that are more appropriate to fulfill the needs of use cases such as "Lightweight VMs", "Virtualization-isolated Containers" or "Trusted Execution Environments (TEEs)". And this, out of necessity, implies extending both libkrun and the kernel bundled with it.
Now, I do understand why the actual kernel shipped with Fedora, the one that will be used by booting thousands of machines and will be running in privileged mode, needs to be held to sky-high standards, which may also include the restriction to not include additional patches, unless it's strongly justified. But, I also think that the kernel bundled in libkrunfw shouldn't be held by such standards.
The kernel bundled in libkrunfw can and only will be used by libkrun to bring up the Virtualized environment where the isolated workload will run. It will only be used in a Virtualization-isolated context, usually by unprivileged users, and in most cases in a pre-containerized environment. I argue, and this is a hill I'm willing to die on, that a bug in either the kernel bundled in libkrunfw or in libkrun itself, can't cause any more damage than any other userspace application shipped in Fedora. In fact, the risk is usually lower, given the context described before.
Having librunfw following the same high standards as the main kernel, we could end up going against our ability to provide new features to the users in a timely manner and gather a much needed feedback that's critical to the success of this project. In practice, this inhibits the possibility of using Fedora as the main upstream distribution for libkrun and the projects depending on it. And this would be very unfortunate, given the roots of libkrun
So, for the reasons stated above, I'd would like to reopen this question to see if together we can find a compromise that would work for all of us.
Thanks, Sergio.
[1] https://build.opensuse.org/package/show/openSUSE:Factory/libkrun [2] https://gvisor.dev/ [3] https://www.qemu.org/
On 3/22/21 05:23, Sergio Lopez wrote:
Hi,
Now that libkrun, libkrunfw and krunvm have landed in openSUSE Tumbleweed [1], I think it's a good time to reopen this discussion. I also had plenty of time to think about it, and my thoughts are now clearer, so please let me explain again why bundling a custom kernel with patches is a critical aspect of the libkrun project, and why this doesn't pose a risk for neither the Fedora Kernel Team, nor Fedora as a whole.
I like to define libkrun as "a dynamic library that enables other programs to easily gain Virtualization-based isolation capabilities, with the minimum possible footprint". While it does integrate the functionality of a VMM (Virtual Machine Monitor), in many aspects is closer to gVisor [2] than to QEMU [3], with the difference that gVisor uses a microkernel built for that specific purpose, while libkrun relies on a custom Linux kernel bundled in libkrunfw.
It's in the very nature of the project the need to explore building new bridges that allow the isolated workload running in the VM to communicate with the Host, in ways that are more appropriate to fulfill the needs of use cases such as "Lightweight VMs", "Virtualization-isolated Containers" or "Trusted Execution Environments (TEEs)". And this, out of necessity, implies extending both libkrun and the kernel bundled with it.
Now, I do understand why the actual kernel shipped with Fedora, the one that will be used by booting thousands of machines and will be running in privileged mode, needs to be held to sky-high standards, which may also include the restriction to not include additional patches, unless it's strongly justified. But, I also think that the kernel bundled in libkrunfw shouldn't be held by such standards.
The kernel bundled in libkrunfw can and only will be used by libkrun to bring up the Virtualized environment where the isolated workload will run. It will only be used in a Virtualization-isolated context, usually by unprivileged users, and in most cases in a pre-containerized environment. I argue, and this is a hill I'm willing to die on, that a bug in either the kernel bundled in libkrunfw or in libkrun itself, can't cause any more damage than any other userspace application shipped in Fedora. In fact, the risk is usually lower, given the context described before.
Having librunfw following the same high standards as the main kernel, we could end up going against our ability to provide new features to the users in a timely manner and gather a much needed feedback that's critical to the success of this project. In practice, this inhibits the possibility of using Fedora as the main upstream distribution for libkrun and the projects depending on it. And this would be very unfortunate, given the roots of libkrun
So, for the reasons stated above, I'd would like to reopen this question to see if together we can find a compromise that would work for all of us.
Thanks, Sergio.
[1] https://build.opensuse.org/package/show/openSUSE:Factory/libkrun [2] https://gvisor.dev/ [3] https://www.qemu.org/
I would like to also recommend that this be reconsidered. We would like to make libkrun a first class citizen as a crun based OCI Runtime. We see benefits for it in both on Linux (Fedora) platforms as it provides a easy to use KVM Container, which works easier with the host then Kata. Secondarily with MAC systems, since it provides a light weight way of running containers on a MAC.
In my option, having Fedora's name associated with this innovative technology would be beneficial to Fedora project.
On Mon, Mar 22, 2021 at 10:23:24AM +0100, Sergio Lopez wrote:
So, for the reasons stated above, I'd would like to reopen this question to see if together we can find a compromise that would work for all of us.
This all sounds good to me. What are we blocked on?
Matthew,
On Tue, Mar 23, 2021 at 8:31 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Mar 22, 2021 at 10:23:24AM +0100, Sergio Lopez wrote:
So, for the reasons stated above, I'd would like to reopen this question to see if together we can find a compromise that would work for all of us.
This all sounds good to me. What are we blocked on?
Right now we're blocked on having an agreement with the kernel team in order to proceed with a kernel that is different from the one shipped and used by Fedora. Once we have this approval, I guess the next step would be updating the current documents on what can (or cannot) be shipped, and what's the approach to follow in such cases, and then, finally, we can proceed with packaging libkrunfw.
AFAIU, the approval must come from the Fedora kernel team. If this ML is not exactly the best way, could someone redirect us to the best way to approach the team and recommend the next steps from here?
One note, Dan Walsh's email seems to be blocked waiting for the moderator to approve it, as it didn't show up here.
In any case, Dan mentioned, and I'm quoting here as is: """ I would like to also recommend that this be reconsidered. We would like to make libkrun a first class citizen as a crun based OCI Runtime. We see benefits for it in both on Linux (Fedora) platforms as it provides a easy to use KVM Container, which works easier with the host then Kata. Secondarily with MAC systems, since it provides a light weight way of running containers on a MAC.
In my option, having Fedora's name associated with this innovative technology would be beneficial to Fedora project. """
Best Regards,
On Mon, Mar 22, 2021 at 4:23 AM Sergio Lopez slp@redhat.com wrote:
Hi,
Now that libkrun, libkrunfw and krunvm have landed in openSUSE Tumbleweed [1], I think it's a good time to reopen this discussion. I also had plenty of time to think about it, and my thoughts are now clearer, so please let me explain again why bundling a custom kernel with patches is a critical aspect of the libkrun project, and why this doesn't pose a risk for neither the Fedora Kernel Team, nor Fedora as a whole.
I like to define libkrun as "a dynamic library that enables other programs to easily gain Virtualization-based isolation capabilities, with the minimum possible footprint". While it does integrate the functionality of a VMM (Virtual Machine Monitor), in many aspects is closer to gVisor [2] than to QEMU [3], with the difference that gVisor uses a microkernel built for that specific purpose, while libkrun relies on a custom Linux kernel bundled in libkrunfw.
It's in the very nature of the project the need to explore building new bridges that allow the isolated workload running in the VM to communicate with the Host, in ways that are more appropriate to fulfill the needs of use cases such as "Lightweight VMs", "Virtualization-isolated Containers" or "Trusted Execution Environments (TEEs)". And this, out of necessity, implies extending both libkrun and the kernel bundled with it.
Now, I do understand why the actual kernel shipped with Fedora, the one that will be used by booting thousands of machines and will be running in privileged mode, needs to be held to sky-high standards, which may also include the restriction to not include additional patches, unless it's strongly justified. But, I also think that the kernel bundled in libkrunfw shouldn't be held by such standards.
The kernel bundled in libkrunfw can and only will be used by libkrun to bring up the Virtualized environment where the isolated workload will run. It will only be used in a Virtualization-isolated context, usually by unprivileged users, and in most cases in a pre-containerized environment. I argue, and this is a hill I'm willing to die on, that a bug in either the kernel bundled in libkrunfw or in libkrun itself, can't cause any more damage than any other userspace application shipped in Fedora. In fact, the risk is usually lower, given the context described before.
Having librunfw following the same high standards as the main kernel, we could end up going against our ability to provide new features to the users in a timely manner and gather a much needed feedback that's critical to the success of this project. In practice, this inhibits the possibility of using Fedora as the main upstream distribution for libkrun and the projects depending on it. And this would be very unfortunate, given the roots of libkrun
So, for the reasons stated above, I'd would like to reopen this question to see if together we can find a compromise that would work for all of us.
The kernel in this case is a part of the libkrunfw package? I don't have a problem with this particular use case, it is not a "kernel" package, and would not be mistaken as the proper kernel in any sort of a real context. On a philosophical note, I would like to see some effort of getting the patches required upstream in some manner, as any project relying on external patches long term is just asking for a lot of unnecessary pain. Even with the patches upstreamed though, a separate build is still required, and I believe this solution is acceptable.
Justin
Justin,
On Tue, Mar 23, 2021 at 10:30 PM Justin Forbes jmforbes@linuxtx.org wrote:
On Mon, Mar 22, 2021 at 4:23 AM Sergio Lopez slp@redhat.com wrote:
Hi,
Now that libkrun, libkrunfw and krunvm have landed in openSUSE Tumbleweed [1], I think it's a good time to reopen this discussion. I also had plenty of time to think about it, and my thoughts are now clearer, so please let me explain again why bundling a custom kernel with patches is a critical aspect of the libkrun project, and why this doesn't pose a risk for neither the Fedora Kernel Team, nor Fedora as a whole.
I like to define libkrun as "a dynamic library that enables other programs to easily gain Virtualization-based isolation capabilities, with the minimum possible footprint". While it does integrate the functionality of a VMM (Virtual Machine Monitor), in many aspects is closer to gVisor [2] than to QEMU [3], with the difference that gVisor uses a microkernel built for that specific purpose, while libkrun relies on a custom Linux kernel bundled in libkrunfw.
It's in the very nature of the project the need to explore building new bridges that allow the isolated workload running in the VM to communicate with the Host, in ways that are more appropriate to fulfill the needs of use cases such as "Lightweight VMs", "Virtualization-isolated Containers" or "Trusted Execution Environments (TEEs)". And this, out of necessity, implies extending both libkrun and the kernel bundled with it.
Now, I do understand why the actual kernel shipped with Fedora, the one that will be used by booting thousands of machines and will be running in privileged mode, needs to be held to sky-high standards, which may also include the restriction to not include additional patches, unless it's strongly justified. But, I also think that the kernel bundled in libkrunfw shouldn't be held by such standards.
The kernel bundled in libkrunfw can and only will be used by libkrun to bring up the Virtualized environment where the isolated workload will run. It will only be used in a Virtualization-isolated context, usually by unprivileged users, and in most cases in a pre-containerized environment. I argue, and this is a hill I'm willing to die on, that a bug in either the kernel bundled in libkrunfw or in libkrun itself, can't cause any more damage than any other userspace application shipped in Fedora. In fact, the risk is usually lower, given the context described before.
Having librunfw following the same high standards as the main kernel, we could end up going against our ability to provide new features to the users in a timely manner and gather a much needed feedback that's critical to the success of this project. In practice, this inhibits the possibility of using Fedora as the main upstream distribution for libkrun and the projects depending on it. And this would be very unfortunate, given the roots of libkrun
So, for the reasons stated above, I'd would like to reopen this question to see if together we can find a compromise that would work for all of us.
The kernel in this case is a part of the libkrunfw package?
Sergio will correct me if I'm mistaken, but libkrunfw actually *is* the kernel bundled in a dynamic library. So, yes, the kernel, in this case, will be shipped as libkrunfw and there's absolutely no way it will be mistaken as the "Fedora Kernel".
I don't have a problem with this particular use case, it is not a "kernel" package, and would not be mistaken as the proper kernel in any sort of a real context.
Perfect! With this in mind, I guess we can simply go ahead and file the bug for adding the package to Fedora.
Sergio, I could happily review that. :-)
On a philosophical note, I would like to see some effort of getting the patches required upstream in some manner, as any project relying on external patches long term is just asking for a lot of unnecessary pain. Even with the patches upstreamed though, a separate build is still required, and I believe this solution is acceptable.
Justin, I can say, without any doubt, that we're on the same page here. There's absolutely zero intent to keep patches downstream, which would only generate us a huge maintenance burden.
However, we may have to, in some cases, at least for some time.
Best Regards,
On Tue, Mar 23, 2021 at 10:47:28PM +0100, Fabiano Fidêncio wrote:
I don't have a problem with this particular use case, it is not a "kernel" package, and would not be mistaken as the proper kernel in any sort of a real context.
Perfect! With this in mind, I guess we can simply go ahead and file the bug for adding the package to Fedora.
Cool, that matches with my surface-level reaction too so I'm glad the people who know the details are also on the same page. :)
kernel@lists.fedoraproject.org
-
Daniel Walsh -
Fabiano Fidêncio -
Jiri Benc -
Justin Forbes -
Matthew Miller -
Sergio Lopez