Heads-up: I've taken ownership over the Cloud SIG's planned "Cloud-Friendly Kernel Packaging" (aka "Modular Kernel Packaging for Cloud") Change [0].
(Sorry for cross-posting this on the kernel and cloud lists, but both the kernel team and the cloud SIG need to be involved in this discussion).
Obviously, we wish for modular kernel packaging and Josh Boyer already reached out to the cloud SIG in this regard a while ago when we didn't really know what we want yet [1]. We do now know that somewhat better [2] but details could still require additional discussion. Our motivation is to save space, time and bandwidth (the latter two referring to required updates) as in the cloud, all three equal to costs. And yes, we also target things other than the kernel to save space.
So, in our case hardware drivers are rather unnecessary and the kernel experts might know other ways to shrink the footprint for our limited use cases. The kernel we require supports all primary architectures (i686 and x86_64 right now, ARM likely later) and is able to run on a hypervisor (primarily KVM and Xen but support for ESXi and Hyper-V are becoming crucial in private clouds). Only. No bare-metal. We also require support for Linux Containers (LXC) to enable the use of Docker.
Now, I heard some people screaming when I said HW drivers are unnecessary. Some people will want to make use of PCI passthrough and while I think we don't want to ship the necessary modules by default, they should be easily installable through a separate package. If some more granularity is acceptable (and makes sense), I think one package for SRIOV NICs, one for graphic cards (to enable mathematical computation stuff) and one for everything else PCI passthrough (plus one for all other HW drivers for the non-cloud products) would be totally nice.
What does everybody think about this? Can it be done? How is it best done? What's the timeframe (we'd really like to see this implemented in F21 Beta but obviously the earlier modular kernels can be tested, the better)? Do you require additional input?
Please try to keep the discussion as simple as possible (in terms of us non-kernel hackers should be able to follow it). Just like Josh requested when reaching out to us: "Explain it to me like I'm a child."
Thanks, Sandro
[0] https://fedoraproject.org/wiki/Cloud_Changelist#Change:_Cloud-Friendly_Kerne... [1] https://lists.fedoraproject.org/pipermail/cloud/2013-October/002862.html [2] https://fedoraproject.org/wiki/Cloud_PRD
On Wed, Mar 5, 2014 at 8:48 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
Heads-up: I've taken ownership over the Cloud SIG's planned "Cloud-Friendly Kernel Packaging" (aka "Modular Kernel Packaging for Cloud") Change [0].
(Sorry for cross-posting this on the kernel and cloud lists, but both the kernel team and the cloud SIG need to be involved in this discussion).
Obviously, we wish for modular kernel packaging and Josh Boyer already reached out to the cloud SIG in this regard a while ago when we didn't really know what we want yet [1]. We do now know that somewhat better [2] but details could still require additional discussion. Our motivation is to save space, time and bandwidth (the latter two referring to required updates) as in the cloud, all three equal to costs. And yes, we also target things other than the kernel to save space.
So, in our case hardware drivers are rather unnecessary and the kernel experts might know other ways to shrink the footprint for our limited use cases. The kernel we require supports all primary architectures (i686 and x86_64 right now, ARM likely later) and is able to run on a hypervisor (primarily KVM and Xen but support for ESXi and Hyper-V are becoming crucial in private clouds). Only. No bare-metal. We also require support for Linux Containers (LXC) to enable the use of Docker.
Now, I heard some people screaming when I said HW drivers are unnecessary. Some people will want to make use of PCI passthrough and while I think we don't want to ship the necessary modules by default, they should be easily installable through a separate package. If some more granularity is acceptable (and makes sense), I think one package for SRIOV NICs, one for graphic cards (to enable mathematical computation stuff) and one for everything else PCI passthrough (plus one for all other HW drivers for the non-cloud products) would be totally nice.
What does everybody think about this? Can it be done? How is it best done? What's the timeframe (we'd really like to see this implemented in F21 Beta but obviously the earlier modular kernels can be tested, the better)? Do you require additional input?
Please try to keep the discussion as simple as possible (in terms of us non-kernel hackers should be able to follow it). Just like Josh requested when reaching out to us: "Explain it to me like I'm a child."
Some drivers are built inside the kernel (for various reasons) so if we keep them in they can't really be modular. If we are going down this route (ex. kernel-hw-drivers sub package) we should have a way to handle upgrades. You don't want to upgrade and be left out without any hardware support because it is now a sub package that nothing pulls in.
How much are you trying to save by this? All modules (excluding extra which is already a sub package) are ~115MB here on x86_64.
On Wed, Mar 5, 2014 at 6:58 PM, drago01 drago01@gmail.com wrote:
On Wed, Mar 5, 2014 at 8:48 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
Heads-up: I've taken ownership over the Cloud SIG's planned "Cloud-Friendly Kernel Packaging" (aka "Modular Kernel Packaging for Cloud") Change [0].
(Sorry for cross-posting this on the kernel and cloud lists, but both the kernel team and the cloud SIG need to be involved in this discussion).
Obviously, we wish for modular kernel packaging and Josh Boyer already reached out to the cloud SIG in this regard a while ago when we didn't really know what we want yet [1]. We do now know that somewhat better [2] but details could still require additional discussion. Our motivation is to save space, time and bandwidth (the latter two referring to required updates) as in the cloud, all three equal to costs. And yes, we also target things other than the kernel to save space.
So, in our case hardware drivers are rather unnecessary and the kernel experts might know other ways to shrink the footprint for our limited use cases. The kernel we require supports all primary architectures (i686 and x86_64 right now, ARM likely later) and is able to run on a hypervisor (primarily KVM and Xen but support for ESXi and Hyper-V are becoming crucial in private clouds). Only. No bare-metal. We also require support for Linux Containers (LXC) to enable the use of Docker.
Now, I heard some people screaming when I said HW drivers are unnecessary. Some people will want to make use of PCI passthrough and while I think we don't want to ship the necessary modules by default, they should be easily installable through a separate package. If some more granularity is acceptable (and makes sense), I think one package for SRIOV NICs, one for graphic cards (to enable mathematical computation stuff) and one for everything else PCI passthrough (plus one for all other HW drivers for the non-cloud products) would be totally nice.
What does everybody think about this? Can it be done? How is it best done? What's the timeframe (we'd really like to see this implemented in F21 Beta but obviously the earlier modular kernels can be tested, the better)? Do you require additional input?
Please try to keep the discussion as simple as possible (in terms of us non-kernel hackers should be able to follow it). Just like Josh requested when reaching out to us: "Explain it to me like I'm a child."
Some drivers are built inside the kernel (for various reasons) so if we keep them in they can't really be modular.
If there are reasons they are built-in (and I think currently nothing is being built-in without good reasons), that should not be changed. Only just talking about what is currently a module (or otherwise a separate file - if applicable). The vmlinuz should be kept as-is and stay the same for all products (server, workstation and cloud).
If we are going down this route (ex. kernel-hw-drivers sub package) we should have a way to handle upgrades. You don't want to upgrade and be left out without any hardware support because it is now a sub package that nothing pulls in.
Yes, that's only too true. But it should be possible to guarantee proper upgrades through means of packaging. And of course this will be tested.
How much are you trying to save by this? All modules (excluding extra which is already a sub package) are ~115MB here on x86_64.
As much as is sensibly possible. Every MB counts but we don't want to drive the kernel team members insane. :)
-- Sandro
On Wed, Mar 05, 2014 at 08:25:12PM +0900, Sandro "red" Mathys wrote:
Some drivers are built inside the kernel (for various reasons) so if we keep them in they can't really be modular.
If there are reasons they are built-in (and I think currently nothing is being built-in without good reasons), that should not be changed. Only just talking about what is currently a module (or otherwise a separate file - if applicable). The vmlinuz should be kept as-is and stay the same for all products (server, workstation and cloud).
If we are going down this route (ex. kernel-hw-drivers sub package) we should have a way to handle upgrades. You don't want to upgrade and be left out without any hardware support because it is now a sub package that nothing pulls in.
Yes, that's only too true. But it should be possible to guarantee proper upgrades through means of packaging. And of course this will be tested.
How much are you trying to save by this? All modules (excluding extra which is already a sub package) are ~115MB here on x86_64.
As much as is sensibly possible. Every MB counts but we don't want to drive the kernel team members insane. :)
I disagree with that approach. 'Every MB counts' is a difficult concept to implement and forces folks (like the kernel) to implement radical changes to achieve some 'pie in the sky' requirement that you are never really sure you have achieved.
How about something more sensible, hard limits. Go through your package requirements and allocate a chunk of space to everyone and see how that works (well not to each userspace package individually but collectively). If that is too big trim some places.
For example, lets start with 100MB package requirement for the kernel (and say 2 GB for userspace). This way the kernel team can implement reasonable changes and monitor proper usage (because things grow over time).
If later on you realize 100 MB is not competitive enough, come back and chop it down to say 50 MB and let the kernel team figure it out.
But please do not come in here with a 'every MB counts' approach. It is not very sustainable for future growth nor really easy to implement from an engineering approach.
Is that acceptable? The kernel team can start with a hard limit of 100MB package requirement (or something reasonably less)? Let's work off budget requirements please.
Cheers, Don
On Wed, Mar 5, 2014 at 9:54 AM, Don Zickus dzickus@redhat.com wrote:
On Wed, Mar 05, 2014 at 08:25:12PM +0900, Sandro "red" Mathys wrote: For example, lets start with 100MB package requirement for the kernel (and say 2 GB for userspace). This way the kernel team can implement reasonable changes and monitor proper usage (because things grow over time).
If later on you realize 100 MB is not competitive enough, come back and chop it down to say 50 MB and let the kernel team figure it out.
But please do not come in here with a 'every MB counts' approach. It is not very sustainable for future growth nor really easy to implement from an engineering approach.
Is that acceptable? The kernel team can start with a hard limit of 100MB package requirement (or something reasonably less)? Let's work off budget requirements please.
This is a fair point. To be honest, I've ignored the "every MB counts" aspect entirely for now. I've instead been focusing on required functionality, because that's likely going to be the main driver of what the resulting size will be.
FWIW, the existing kernel package installed today (a debug kernel even) is ~142 MB. 123MB of that is the /lib/modules content. ~6MB of that is vmlinuz. The remaining 13MB is the initramfs, which is actually something that composes on the system during install and not something we can shrink from a packaging standpoint.
josh
Am 05.03.2014 16:02, schrieb Josh Boyer:
FWIW, the existing kernel package installed today (a debug kernel even) is ~142 MB. 123MB of that is the /lib/modules content. ~6MB of that is vmlinuz. The remaining 13MB is the initramfs, which is actually something that composes on the system during install and not something we can shrink from a packaging standpoint
honestly "glibc-common" would be more useful and less critical to split into subpackages
* missing a locale and fallback to english leaves the option install whatever package - i would not miss anything than "de" and "en" on all machines i maintain - cloud or not
* missing a kernel module and refuse to boot is critical
because of that CC to @devel ________________________________________________________
114,08 MB glibc-common 132,60 MB kernel
[harry@rh]$ rpm -q --filesbypkg glibc-common | grep locale | wc -l 419
[harry@rh]$ rpm -q --filesbypkg glibc-common | grep -v locale | wc -l 257
On Wed, Mar 05, 2014 at 10:02:17AM -0500, Josh Boyer wrote:
On Wed, Mar 5, 2014 at 9:54 AM, Don Zickus dzickus@redhat.com wrote:
On Wed, Mar 05, 2014 at 08:25:12PM +0900, Sandro "red" Mathys wrote: For example, lets start with 100MB package requirement for the kernel (and say 2 GB for userspace). This way the kernel team can implement reasonable changes and monitor proper usage (because things grow over time).
If later on you realize 100 MB is not competitive enough, come back and chop it down to say 50 MB and let the kernel team figure it out.
But please do not come in here with a 'every MB counts' approach. It is not very sustainable for future growth nor really easy to implement from an engineering approach.
Is that acceptable? The kernel team can start with a hard limit of 100MB package requirement (or something reasonably less)? Let's work off budget requirements please.
This is a fair point. To be honest, I've ignored the "every MB counts" aspect entirely for now. I've instead been focusing on required functionality, because that's likely going to be the main driver of what the resulting size will be.
Of course. :-)
FWIW, the existing kernel package installed today (a debug kernel even) is ~142 MB. 123MB of that is the /lib/modules content. ~6MB of that is vmlinuz. The remaining 13MB is the initramfs, which is actually something that composes on the system during install and not something we can shrink from a packaging standpoint.
It also helps with monitoring. 3-4 years from now after all the chopping, these pacakges bloat right back up and everyone forgets why we chopped in the first place. Hard requirements help keep everything in check and forces people to request more space which the cloud team can evaluate properly and still control their enviroment.
Anyway, this was just my rant for the day. I saw the initial email and immediately disagreed with the approach. Spent a few minutes thinking of a better way and thought, well I wasted a few minutes thinking about it might as well respond in email. :-)
Cheers, Don
On Thu, Mar 6, 2014 at 12:13 AM, Don Zickus dzickus@redhat.com wrote:
On Wed, Mar 05, 2014 at 10:02:17AM -0500, Josh Boyer wrote:
On Wed, Mar 5, 2014 at 9:54 AM, Don Zickus dzickus@redhat.com wrote:
On Wed, Mar 05, 2014 at 08:25:12PM +0900, Sandro "red" Mathys wrote: For example, lets start with 100MB package requirement for the kernel (and say 2 GB for userspace). This way the kernel team can implement reasonable changes and monitor proper usage (because things grow over time).
If later on you realize 100 MB is not competitive enough, come back and chop it down to say 50 MB and let the kernel team figure it out.
But please do not come in here with a 'every MB counts' approach. It is not very sustainable for future growth nor really easy to implement from an engineering approach.
Is that acceptable? The kernel team can start with a hard limit of 100MB package requirement (or something reasonably less)? Let's work off budget requirements please.
This is a fair point. To be honest, I've ignored the "every MB counts" aspect entirely for now. I've instead been focusing on required functionality, because that's likely going to be the main driver of what the resulting size will be.
That's the point, we want a reasonably small package while still providing the required functionality. Not sure how providing a fixed size number is helping in this. But most of all, I didn't throw in a number because I have no idea what is reasonably possible. I really only just said "every MB counts" because the question came up before (in Josh's old thread) and I hoped I could stop this discussion from happening again before we have any numbers for this.
Of course. :-)
FWIW, the existing kernel package installed today (a debug kernel even) is ~142 MB. 123MB of that is the /lib/modules content. ~6MB of that is vmlinuz. The remaining 13MB is the initramfs, which is actually something that composes on the system during install and not something we can shrink from a packaging standpoint.
It also helps with monitoring. 3-4 years from now after all the chopping, these pacakges bloat right back up and everyone forgets why we chopped in the first place. Hard requirements help keep everything in check and forces people to request more space which the cloud team can evaluate properly and still control their enviroment.
Well, if we can remember why we put up a fixed size requirement, why can't we remember why we did the chopping? ;) Anyway, I think it's fair to define a "kernel-core should be smaller than X MB" requirement but I don't think it's fair to say Y MB because I like the number Y. I also don't like that we might throw out e.g. NFS just because we're 1MB over the limit.
But if it helps the kernel team to have a fixed number, someone tell me what we roughly save by throwing out the stuff we discussed and we can discuss what number would long-termish make sense, I guess. Also, I'm not sure whether we should measure the extracted files, the compressed RPM or both.
-- Sandro
On Thu, Mar 06, 2014 at 08:16:00AM +0900, Sandro "red" Mathys wrote:
On Thu, Mar 6, 2014 at 12:13 AM, Don Zickus dzickus@redhat.com wrote:
On Wed, Mar 05, 2014 at 10:02:17AM -0500, Josh Boyer wrote:
On Wed, Mar 5, 2014 at 9:54 AM, Don Zickus dzickus@redhat.com wrote:
On Wed, Mar 05, 2014 at 08:25:12PM +0900, Sandro "red" Mathys wrote: For example, lets start with 100MB package requirement for the kernel (and say 2 GB for userspace). This way the kernel team can implement reasonable changes and monitor proper usage (because things grow over time).
If later on you realize 100 MB is not competitive enough, come back and chop it down to say 50 MB and let the kernel team figure it out.
But please do not come in here with a 'every MB counts' approach. It is not very sustainable for future growth nor really easy to implement from an engineering approach.
Is that acceptable? The kernel team can start with a hard limit of 100MB package requirement (or something reasonably less)? Let's work off budget requirements please.
This is a fair point. To be honest, I've ignored the "every MB counts" aspect entirely for now. I've instead been focusing on required functionality, because that's likely going to be the main driver of what the resulting size will be.
That's the point, we want a reasonably small package while still providing the required functionality. Not sure how providing a fixed size number is helping in this. But most of all, I didn't throw in a number because I have no idea what is reasonably possible. I really only just said "every MB counts" because the question came up before (in Josh's old thread) and I hoped I could stop this discussion from happening again before we have any numbers for this.
Ever work in the embedded space? Every MB counts there too. :-) This was solved by creating budgets for size and memory requirements. This helped control bloat, which is going to be your biggest problem with cloud deployment.
What concerns me is that you don't know what size your cloud deployment is but expect everyone to just chop chop chop. How do we know if the kernel is already at the right size?
There is s a huge difference between re-architecting the kernel packaging to save 1 or 2 MB (off ~143 MB size currently) vs. re-architecting to save 50 MB. The former is really a wasted exercise in the bigger picture, wherease the latter (if proven needed) accomplishes something.
But again it comes down to understanding your environment. Understanding your environment revovles around control. I get the impression you are not sure what size your environment should be.
So I was proposing the kernel stay put or maybe create _one_ extras package that gets installed in addition to the bzImage. But from the sound of it, the chopping is really going to get you savings of about ~30MB or so.
The thing is a few years ago, the kernel converted a lot of modules to built-ins (inside the kernel). What that means from a novice perspective is we took a whole bunch of external bloat that could be stripped away and stuffed it into the kernel binary itself. The goal at the time was to speed up boot times (because loading modules was slow).
Now with the re-design of module loading in the last couple of years, maybe this can be reverted. This could shave MBs off the kernel binary itself. And then you can only package the modules cloud really needs.
This at least has the ability to scale across other SIGs too.
However, this all hinges on _how much chopping we should do_. I doubt Josh really wants to embark on this thrashing without very good convincing reason to do so.
So having some numbers to work off of, provides us the right idea how much and what type of work needs to get done (little tweaks vs re-think the whole approach).
Of course. :-)
FWIW, the existing kernel package installed today (a debug kernel even) is ~142 MB. 123MB of that is the /lib/modules content. ~6MB of that is vmlinuz. The remaining 13MB is the initramfs, which is actually something that composes on the system during install and not something we can shrink from a packaging standpoint.
It also helps with monitoring. 3-4 years from now after all the chopping, these pacakges bloat right back up and everyone forgets why we chopped in the first place. Hard requirements help keep everything in check and forces people to request more space which the cloud team can evaluate properly and still control their enviroment.
Well, if we can remember why we put up a fixed size requirement, why can't we remember why we did the chopping? ;) Anyway, I think it's
Heh. Ever work with open source projects? The turnover and lost knowledge is one thing. However, to me the biggets problem would be the 'lack of caring'.
Trust me you can convince everyone to chop MBs out of their packages. You might even get something really small for your cloud deployment.
Fast forward a year. Do you think package maintainers are going to remember to go through the painful exercise of chopping again when they rebase their packages or add new features? Probably not. Bloat will shoot right back up.
Heck look at Fedora in the last few years. It went from running fine on a laptop with 2GB of memory to now needing a laptop with 4GB of memory and lots of disk space. Why? Because everyone thinks memory and disk is cheap.
However, cloud folks have a different attitude. Memory and disk is _not_ cheap and can impact their business.
Without disk and memory budgets how are you going to manage the bloat and make sure the cloud deployments are competitive and provide the host providers the most bang for the resource buck?
And I get that it is hard to come up with numbers like this. So I would start with an image and see how it compares. I think Cloud already has an image. Is it to big? What makes you believe that? Are competitors that much smaller?
Then start chopping say 5% off at time to see how low you can bring it before everyone screams too much.
fair to define a "kernel-core should be smaller than X MB" requirement but I don't think it's fair to say Y MB because I like the number Y. I also don't like that we might throw out e.g. NFS just because we're 1MB over the limit.
Why not? If the host provider can buy a 1 TB disk and carve out X number of cloud deployments on there, then if Fedora Cloud bloats up 10%, that means less cloud deployments and less revenue. So yes, I would be concerned over the 1MB. In fact, I would come back and ask the kernel team to find something else to chop to bring it down in size. Or look to see if some other package can trade the size.
But if it helps the kernel team to have a fixed number, someone tell me what we roughly save by throwing out the stuff we discussed and we can discuss what number would long-termish make sense, I guess. Also, I'm not sure whether we should measure the extracted files, the compressed RPM or both.
It should be the opposite. Tell us what number makes long term sense to you and we can figure out what to throw out. I would calculate the number based on deployment size (that is ultimately what you need to manage). That includes the kernel binary, the initrd and the /lib/modules (though the initrd image is thrown out once the disk is mounted).
I can also tell you how much memory that will take up too (as I did a similar analysis to reduce the kdump memory usage on RHEL).
Based on Josh's rough number of ~143MB, I would selfishly propose 150 MB disk space requirement. :-) The memory requirement for that is going to be about 300MB I believe (I would have to re-run the numbers and figure out what modules are loaded [probably a _lot_ less than that though]).
Cheers, Don
On Thu, Mar 6, 2014 at 9:57 AM, Don Zickus dzickus@redhat.com wrote:
On Thu, Mar 06, 2014 at 08:16:00AM +0900, Sandro "red" Mathys wrote:
That's the point, we want a reasonably small package while still providing the required functionality. Not sure how providing a fixed size number is helping in this. But most of all, I didn't throw in a number because I have no idea what is reasonably possible. I really only just said "every MB counts" because the question came up before (in Josh's old thread) and I hoped I could stop this discussion from happening again before we have any numbers for this.
Ever work in the embedded space? Every MB counts there too. :-) This was solved by creating budgets for size and memory requirements. This helped control bloat, which is going to be your biggest problem with cloud deployment.
What concerns me is that you don't know what size your cloud deployment is but expect everyone to just chop chop chop. How do we know if the kernel is already at the right size?
There is s a huge difference between re-architecting the kernel packaging to save 1 or 2 MB (off ~143 MB size currently) vs. re-architecting to save 50 MB. The former is really a wasted exercise in the bigger picture, wherease the latter (if proven needed) accomplishes something.
But again it comes down to understanding your environment. Understanding your environment revovles around control. I get the impression you are not sure what size your environment should be.
So I was proposing the kernel stay put or maybe create _one_ extras package that gets installed in addition to the bzImage. But from the
Right. When I said I had kernel-core and kernel-drivers, I wasn't being theoretical. I already did the work in the spec file to split it into kernel-core and kernel-drivers. The kernel package becomes a metapackage that requires the other two, so that existing installs and anaconda don't have to change (assuming I did thinks correctly). Cloud can just specify kernel-core in the kickstart or whatever.
sound of it, the chopping is really going to get you savings of about ~30MB or so.
I spent some time yesterday hacking around on an existing VM and just removing stuff from /lib/modules/ for an installed kernel. I was able to get it down from 123MB to 58MB by axing entire subsystems that clearly didn't apply and running depmod on the results to make sure there weren't missing dependencies. Some stuff had to be added back (want virtio_scsi? you need target and libfc), but a lot could be removed. That brings the total to about 81MB for vmlinuz, initramfs, and /lib/modules for that particular kernel.
In those 81MB, I still had all of the main GPU drivers, all of the intel and a few other ethernet drivers, ext4, xfs, btrfs, nfs, the vast majority of the networking modules (so iptables and netfilter), scsi, acpi, block, char, etc. The major things missing were bluetooth and wireless, infiniband, some firewire stuff. Basically it resulted in a system that boots perfectly fine in a VM for a variety of different use cases.
I think that's a reasonable start, and it's a significant reduction. Beyond that, we get into much less reduced savings and having to move stuff around on a finer level. For the curious, I uploaded the module list here:
http://jwboyer.fedorapeople.org/pub/modules.small
Again, this was just hacking around on an installed system. Still work to do at a packaging level, but this is as good as anything to start from.
josh
On Thu, Mar 06, 2014 at 10:33:47AM -0500, Josh Boyer wrote:
On Thu, Mar 6, 2014 at 9:57 AM, Don Zickus dzickus@redhat.com wrote:
On Thu, Mar 06, 2014 at 08:16:00AM +0900, Sandro "red" Mathys wrote:
That's the point, we want a reasonably small package while still providing the required functionality. Not sure how providing a fixed size number is helping in this. But most of all, I didn't throw in a number because I have no idea what is reasonably possible. I really only just said "every MB counts" because the question came up before (in Josh's old thread) and I hoped I could stop this discussion from happening again before we have any numbers for this.
Ever work in the embedded space? Every MB counts there too. :-) This was solved by creating budgets for size and memory requirements. This helped control bloat, which is going to be your biggest problem with cloud deployment.
What concerns me is that you don't know what size your cloud deployment is but expect everyone to just chop chop chop. How do we know if the kernel is already at the right size?
There is s a huge difference between re-architecting the kernel packaging to save 1 or 2 MB (off ~143 MB size currently) vs. re-architecting to save 50 MB. The former is really a wasted exercise in the bigger picture, wherease the latter (if proven needed) accomplishes something.
But again it comes down to understanding your environment. Understanding your environment revovles around control. I get the impression you are not sure what size your environment should be.
So I was proposing the kernel stay put or maybe create _one_ extras package that gets installed in addition to the bzImage. But from the
Right. When I said I had kernel-core and kernel-drivers, I wasn't being theoretical. I already did the work in the spec file to split it into kernel-core and kernel-drivers. The kernel package becomes a metapackage that requires the other two, so that existing installs and anaconda don't have to change (assuming I did thinks correctly). Cloud can just specify kernel-core in the kickstart or whatever.
sound of it, the chopping is really going to get you savings of about ~30MB or so.
I spent some time yesterday hacking around on an existing VM and just removing stuff from /lib/modules/ for an installed kernel. I was able to get it down from 123MB to 58MB by axing entire subsystems that clearly didn't apply and running depmod on the results to make sure there weren't missing dependencies. Some stuff had to be added back (want virtio_scsi? you need target and libfc), but a lot could be removed. That brings the total to about 81MB for vmlinuz, initramfs, and /lib/modules for that particular kernel.
Ok. That is pretty damn good. Good job! Again the question is, is that enough?
In those 81MB, I still had all of the main GPU drivers, all of the intel and a few other ethernet drivers, ext4, xfs, btrfs, nfs, the vast majority of the networking modules (so iptables and netfilter), scsi, acpi, block, char, etc. The major things missing were bluetooth and wireless, infiniband, some firewire stuff. Basically it resulted in a system that boots perfectly fine in a VM for a variety of different use cases.
I think that's a reasonable start, and it's a significant reduction. Beyond that, we get into much less reduced savings and having to move stuff around on a finer level. For the curious, I uploaded the module list here:
You might be able to chop sound/ too (I know we could probably go crazy and chop lots of other things too., just thought sound/ would be a big easy chop).
Does this approach affect other SIGs too? Most of the others don't care about disk and memory space, right?
Cheers, Don
On Thu, Mar 06, 2014 at 10:33:47AM -0500, Josh Boyer wrote:
Right. When I said I had kernel-core and kernel-drivers, I wasn't being theoretical. I already did the work in the spec file to split it into kernel-core and kernel-drivers. The kernel package becomes a metapackage that requires the other two, so that existing installs and anaconda don't have to change (assuming I did thinks correctly). Cloud can just specify kernel-core in the kickstart or whatever.
Does yum's kernel-handling magic need to change to handle this? Probably you have already thought of that.
I think that's a reasonable start, and it's a significant reduction. Beyond that, we get into much less reduced savings and having to move stuff around on a finer level. For the curious, I uploaded the module list here:
+1 to getting the low-hanging fruit and not worrying too much about the stuff up on the hard to reach branches.
On Thu, Mar 6, 2014 at 11:18 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Thu, Mar 06, 2014 at 10:33:47AM -0500, Josh Boyer wrote:
Right. When I said I had kernel-core and kernel-drivers, I wasn't being theoretical. I already did the work in the spec file to split it into kernel-core and kernel-drivers. The kernel package becomes a metapackage that requires the other two, so that existing installs and anaconda don't have to change (assuming I did thinks correctly). Cloud can just specify kernel-core in the kickstart or whatever.
Does yum's kernel-handling magic need to change to handle this? Probably you have already thought of that.
I gave it some thought. I haven't tested anything. Existing magic should work for most installs since there's still a "kernel" package and that's what yum keys off of. If/when we do this, I'll certainly test more carefully to make sure.
That brings up a question though, how often would cloud expect to do a yum update of individual packages as opposed to just updating the entire image? If we expect the 3 kernel magic to work there, then some changes to yum/dnf may be required given that Cloud would be explicitly specifying "kernel-core" and not "kernel".
josh
Am 06.03.2014 19:14, schrieb Josh Boyer:
That brings up a question though, how often would cloud expect to do a yum update of individual packages as opposed to just updating the entire image?
in the best case you do the same as i do on our "private cloud" install once and then upgrade for 5,6...10 years i consider anything which forces me to a blank re-install as broken
On Thu, Mar 06, 2014 at 01:14:54PM -0500, Josh Boyer wrote:
That brings up a question though, how often would cloud expect to do a yum update of individual packages as opposed to just updating the entire image? If we expect the 3 kernel magic to work there, then
Unless we jump into Colin's rpm ostree future very quickly, we need to support this.
some changes to yum/dnf may be required given that Cloud would be explicitly specifying "kernel-core" and not "kernel".
There is a hardcoded default list for install-instead-of-upgrade packages -- I think we'd just ask for kernel-core to be added to that. Additonally, the protected packages functionality (which keeps you from removing the running kernel) may need a tweak.
On Thu, Mar 6, 2014 at 1:48 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Thu, Mar 06, 2014 at 01:14:54PM -0500, Josh Boyer wrote:
That brings up a question though, how often would cloud expect to do a yum update of individual packages as opposed to just updating the entire image? If we expect the 3 kernel magic to work there, then
Unless we jump into Colin's rpm ostree future very quickly, we need to support this.
OK.
some changes to yum/dnf may be required given that Cloud would be explicitly specifying "kernel-core" and not "kernel".
There is a hardcoded default list for install-instead-of-upgrade packages -- I think we'd just ask for kernel-core to be added to that. Additonally, the protected packages functionality (which keeps you from removing the running kernel) may need a tweak.
Sigh. I was trying to keep it as self-contained to the kernel package as possible. Having to modify a bunch of other packages increases the test matrix and the hassle factor. Oh well.
josh
On Fri, Mar 7, 2014 at 3:53 AM, Josh Boyer jwboyer@fedoraproject.org wrote:
On Thu, Mar 6, 2014 at 1:48 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Thu, Mar 06, 2014 at 01:14:54PM -0500, Josh Boyer wrote:
That brings up a question though, how often would cloud expect to do a yum update of individual packages as opposed to just updating the entire image? If we expect the 3 kernel magic to work there, then
Unless we jump into Colin's rpm ostree future very quickly, we need to support this.
OK.
some changes to yum/dnf may be required given that Cloud would be explicitly specifying "kernel-core" and not "kernel".
There is a hardcoded default list for install-instead-of-upgrade packages -- I think we'd just ask for kernel-core to be added to that. Additonally, the protected packages functionality (which keeps you from removing the running kernel) may need a tweak.
Sigh. I was trying to keep it as self-contained to the kernel package as possible. Having to modify a bunch of other packages increases the test matrix and the hassle factor. Oh well.
What about Anaconda? I guess it does have its own mechanic to guarantee a kernel is installed, right? Probably hardcoded as well. Since we're going to build future images with ImageFactory/Anaconda, it must be possible to install either kernel-core or kernel (when installing with a kickstart file, that is). I think putting "-kernel" in %packages doesn't currently have much effect, does it?
So if I'm right and Anaconda needs to be adapted for this, I'm happy to reach out to the Anaconda team on our behalf, if necessary.
-- Sandro
On Fri, Mar 07, 2014 at 04:28:56PM +0900, Sandro red Mathys wrote:
What about Anaconda? I guess it does have its own mechanic to guarantee a kernel is installed, right? Probably hardcoded as well. Since we're going to build future images with ImageFactory/Anaconda, it must be possible to install either kernel-core or kernel (when installing with a kickstart file, that is). I think putting "-kernel" in %packages doesn't currently have much effect, does it?
So if I'm right and Anaconda needs to be adapted for this, I'm happy to reach out to the Anaconda team on our behalf, if necessary.
Yeah. We also need to have it not install the kernel (and there's no need to mess with a bootloader) for docker images, so that all kind of comes together.
On Fri, Mar 7, 2014 at 8:14 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Mar 07, 2014 at 04:28:56PM +0900, Sandro red Mathys wrote:
What about Anaconda? I guess it does have its own mechanic to guarantee a kernel is installed, right? Probably hardcoded as well. Since we're going to build future images with ImageFactory/Anaconda, it must be possible to install either kernel-core or kernel (when installing with a kickstart file, that is). I think putting "-kernel" in %packages doesn't currently have much effect, does it?
So if I'm right and Anaconda needs to be adapted for this, I'm happy to reach out to the Anaconda team on our behalf, if necessary.
Yeah. We also need to have it not install the kernel (and there's no need to mess with a bootloader) for docker images, so that all kind of comes together.
I'm beginning to like this whole idea less and less.
josh
On Fri, Mar 07, 2014 at 08:23:56AM -0500, Josh Boyer wrote:
So if I'm right and Anaconda needs to be adapted for this, I'm happy to reach out to the Anaconda team on our behalf, if necessary.
Yeah. We also need to have it not install the kernel (and there's no need to mess with a bootloader) for docker images, so that all kind of comes together.
I'm beginning to like this whole idea less and less.
Actually, the part I said should make you less worried about that, I hope. We _already_ need that change completely independently of the work you are doing here, so there is no additional cost.
On Fri, Mar 7, 2014 at 3:14 AM, Josh Boyer jwboyer@fedoraproject.org wrote:
On Thu, Mar 6, 2014 at 11:18 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Thu, Mar 06, 2014 at 10:33:47AM -0500, Josh Boyer wrote:
Right. When I said I had kernel-core and kernel-drivers, I wasn't being theoretical. I already did the work in the spec file to split it into kernel-core and kernel-drivers. The kernel package becomes a metapackage that requires the other two, so that existing installs and anaconda don't have to change (assuming I did thinks correctly). Cloud can just specify kernel-core in the kickstart or whatever.
Does yum's kernel-handling magic need to change to handle this? Probably you have already thought of that.
I gave it some thought. I haven't tested anything. Existing magic should work for most installs since there's still a "kernel" package and that's what yum keys off of. If/when we do this, I'll certainly test more carefully to make sure.
That brings up a question though, how often would cloud expect to do a yum update of individual packages as opposed to just updating the entire image? If we expect the 3 kernel magic to work there, then some changes to yum/dnf may be required given that Cloud would be explicitly specifying "kernel-core" and not "kernel".
Yesterday, I updated to Josh's 2.5 kernel, then I removed kernel and kernel-drivers, only leaving kernel-core. Today, I again updates, this time 3.8 being available. Now guess what? The "install instead of update" magic worked. I know have both kernel-core packages. So I figure yum works this magic with any kernel* package.
So it would seem no change to yum is necessary after all. Going to reach out to the yum developers to find out the details. Also, whether dnf does the exact same or not.
-- Sandro
On Fri, Mar 14, 2014 at 3:46 PM, Sandro "red" Mathys red@fedoraproject.org wrote:
On Fri, Mar 7, 2014 at 3:14 AM, Josh Boyer jwboyer@fedoraproject.org wrote:
On Thu, Mar 6, 2014 at 11:18 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Thu, Mar 06, 2014 at 10:33:47AM -0500, Josh Boyer wrote:
Right. When I said I had kernel-core and kernel-drivers, I wasn't being theoretical. I already did the work in the spec file to split it into kernel-core and kernel-drivers. The kernel package becomes a metapackage that requires the other two, so that existing installs and anaconda don't have to change (assuming I did thinks correctly). Cloud can just specify kernel-core in the kickstart or whatever.
Does yum's kernel-handling magic need to change to handle this? Probably you have already thought of that.
I gave it some thought. I haven't tested anything. Existing magic should work for most installs since there's still a "kernel" package and that's what yum keys off of. If/when we do this, I'll certainly test more carefully to make sure.
That brings up a question though, how often would cloud expect to do a yum update of individual packages as opposed to just updating the entire image? If we expect the 3 kernel magic to work there, then some changes to yum/dnf may be required given that Cloud would be explicitly specifying "kernel-core" and not "kernel".
Yesterday, I updated to Josh's 2.5 kernel, then I removed kernel and kernel-drivers, only leaving kernel-core. Today, I again updates, this time 3.8 being available. Now guess what? The "install instead of update" magic worked. I know have both kernel-core packages. So I figure yum works this magic with any kernel* package.
So it would seem no change to yum is necessary after all. Going to reach out to the yum developers to find out the details. Also, whether dnf does the exact same or not.
Okay, I poked around in the yum source, yum docs and kernel packages a bit. So yum (and some testing confirms, dnf too) does not check the package names but the provides (obviously, thinking about it). The actual magic being that both kernel and kernel-core provide "kernel". If that is still the case once the patch is merged from copr to Fedora, no changes to yum or dnf should become necessary. Which would probably leave us with "anaconda must allow for installing only just kernel-core instead of kernel (when kickstarting)" as the only necessary change to Fedora. I guess.
-- Sandro
On Fri, Mar 14, 2014 at 16:42:11 +0900, "Sandro "red" Mathys" red@fedoraproject.org wrote:
Okay, I poked around in the yum source, yum docs and kernel packages a bit. So yum (and some testing confirms, dnf too) does not check the package names but the provides (obviously, thinking about it). The actual magic being that both kernel and kernel-core provide "kernel". If that is still the case once the patch is merged from copr to Fedora, no changes to yum or dnf should become necessary. Which would probably leave us with "anaconda must allow for installing only just kernel-core instead of kernel (when kickstarting)" as the only necessary change to Fedora. I guess.
The behavior is controlled by the installonlypkgs setting in yum.conf. From the man page:
installonlypkgs List of package provides that should only ever be installed, never updated. Kernels in particular fall into this category. Defaults to kernel, kernel-bigmem, kernel-enter‐ prise, kernel-smp, kernel-modules, kernel-debug, kernel-unsup‐ ported, kernel-source, kernel-devel, kernel-PAE, kernel-PAE- debug.
I believe kernel-modules-extra also got added to the default relatively recently. But that change doesn't seem to have made it to the documentation. (That package provides installonlypkgs(kernel-module), so there might be some extra magic related to that provide.)
On Fri, Mar 14, 2014 at 11:00 PM, Bruno Wolff III bruno@wolff.to wrote:
On Fri, Mar 14, 2014 at 16:42:11 +0900,
"Sandro "red" Mathys" red@fedoraproject.org wrote:
Okay, I poked around in the yum source, yum docs and kernel packages a bit. So yum (and some testing confirms, dnf too) does not check the package names but the provides (obviously, thinking about it). The actual magic being that both kernel and kernel-core provide "kernel". If that is still the case once the patch is merged from copr to Fedora, no changes to yum or dnf should become necessary. Which would probably leave us with "anaconda must allow for installing only just kernel-core instead of kernel (when kickstarting)" as the only necessary change to Fedora. I guess.
The behavior is controlled by the installonlypkgs setting in yum.conf. From the man page:
installonlypkgs List of package provides that should only ever be installed, never updated. Kernels in particular fall into this category. Defaults to kernel, kernel-bigmem, kernel-enter‐ prise, kernel-smp, kernel-modules, kernel-debug, kernel-unsup‐ ported, kernel-source, kernel-devel, kernel-PAE, kernel-PAE- debug.
I believe kernel-modules-extra also got added to the default relatively recently. But that change doesn't seem to have made it to the documentation. (That package provides installonlypkgs(kernel-module), so there might be some extra magic related to that provide.)
kernel-modules-extra hasn't been added, but installonlypkgs(kernel-module) has been. Well, at least that's what I see in the current code on git. And kernel-drivers also provides installonlypkgs(kernel-module).
-- Sandro
On Fri, Mar 14, 2014 at 03:46:02PM +0900, Sandro red Mathys wrote:
Yesterday, I updated to Josh's 2.5 kernel, then I removed kernel and kernel-drivers, only leaving kernel-core. Today, I again updates, this time 3.8 being available. Now guess what? The "install instead of update" magic worked. I know have both kernel-core packages. So I figure yum works this magic with any kernel* package.
And this is why I should read all the mail before responding. :)
How about the removal protection?
On Sat, Mar 15, 2014 at 12:06 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Mar 14, 2014 at 03:46:02PM +0900, Sandro red Mathys wrote:
Yesterday, I updated to Josh's 2.5 kernel, then I removed kernel and kernel-drivers, only leaving kernel-core. Today, I again updates, this time 3.8 being available. Now guess what? The "install instead of update" magic worked. I know have both kernel-core packages. So I figure yum works this magic with any kernel* package.
And this is why I should read all the mail before responding. :)
How about the removal protection?
Oh, good catch. Also, that's a funny one.
So, you can't remove the currently running kernel-core. As expected.
BUT, you can remove the currently running kernel/kernel-drivers. That's good because we want those packages to be optionally (un)installable but bad because if you actually require kernel-drivers, well, it's obviously bad.
Not sure if there's a good fix to this. We'd either limit people to only being able to get rid of kernel-drivers with some dancing around with different kernel versions and some rebooting, etc. Or we allow people to remove the kernel-drivers of the running kernel which defeats the purpose of the protection. #leSigh
Oh (#2), and here, dnf actually differs from yum. dnf protects *none* of the packages. So that's definitely a bug and I'll report it once we know exactly what behavior we want (so that yum and dnf will do the same thing).
-- Sandro
Am 14.03.2014 16:25, schrieb Sandro red Mathys:
Oh (#2), and here, dnf actually differs from yum. dnf protects *none* of the packages. So that's definitely a bug and I'll report it once we know exactly what behavior we want (so that yum and dnf will do the same thing)
be careful, on the devel-list that leaded to a flamewar without intention https://lists.fedoraproject.org/pipermail/devel/2014-January/193310.html
On Fri, Mar 14, 2014 at 11:25 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
On Sat, Mar 15, 2014 at 12:06 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Mar 14, 2014 at 03:46:02PM +0900, Sandro red Mathys wrote:
Yesterday, I updated to Josh's 2.5 kernel, then I removed kernel and kernel-drivers, only leaving kernel-core. Today, I again updates, this time 3.8 being available. Now guess what? The "install instead of update" magic worked. I know have both kernel-core packages. So I figure yum works this magic with any kernel* package.
And this is why I should read all the mail before responding. :)
How about the removal protection?
Oh, good catch. Also, that's a funny one.
So, you can't remove the currently running kernel-core. As expected.
BUT, you can remove the currently running kernel/kernel-drivers. That's good because we want those packages to be optionally (un)installable but bad because if you actually require kernel-drivers, well, it's obviously bad.
"Require" is the word that needs further definition.
Not sure if there's a good fix to this. We'd either limit people to only being able to get rid of kernel-drivers with some dancing around with different kernel versions and some rebooting, etc. Or we allow people to remove the kernel-drivers of the running kernel which defeats the purpose of the protection. #leSigh
I don't think we want to over-complicate things here. Removing -drivers from a machine, whether it's running the corresponding kernel or not, shouldn't be a huge deal. They might not have support for some hotplugged device in that case. Also, it's not really going to be a common case for people to do.
Oh (#2), and here, dnf actually differs from yum. dnf protects *none* of the packages. So that's definitely a bug and I'll report it once we know exactly what behavior we want (so that yum and dnf will do the same thing).
That's not a bug. They did that on purpose. There was a big thread about it on the devel list a while ago.
josh
On Sat, Mar 15, 2014 at 12:25:48AM +0900, Sandro red Mathys wrote:
Not sure if there's a good fix to this. We'd either limit people to only being able to get rid of kernel-drivers with some dancing around with different kernel versions and some rebooting, etc. Or we allow people to remove the kernel-drivers of the running kernel which defeats the purpose of the protection. #leSigh
I think the first is preferable; the main case where you get a kernel without the drivers package is when you're building something intended to be small, and going downwards isn't the way to really do that.
Oh (#2), and here, dnf actually differs from yum. dnf protects *none* of the packages. So that's definitely a bug and I'll report it once we know exactly what behavior we want (so that yum and dnf will do the same thing).
Apparently this is by design in dnf; it's one of the features they didn't see as valuable. Which is funny to me because that was the case with yum initially too (we wrote it as a plugin for that reason) but over time it became a core feature).
On Fri, Mar 14, 2014 at 11:59 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Sat, Mar 15, 2014 at 12:25:48AM +0900, Sandro red Mathys wrote:
Not sure if there's a good fix to this. We'd either limit people to only being able to get rid of kernel-drivers with some dancing around with different kernel versions and some rebooting, etc. Or we allow people to remove the kernel-drivers of the running kernel which defeats the purpose of the protection. #leSigh
I think the first is preferable; the main case where you get a kernel without the drivers package is when you're building something intended to be small, and going downwards isn't the way to really do that.
That protection needs to be done in yum/dnf then. I don't see a way to do it cleanly in the kernel packages themselves.
josh
On Fri, Mar 14, 2014 at 12:06:21PM -0400, Josh Boyer wrote:
I think the first is preferable; the main case where you get a kernel without the drivers package is when you're building something intended to be small, and going downwards isn't the way to really do that.
That protection needs to be done in yum/dnf then. I don't see a way to do it cleanly in the kernel packages themselves.
Yeah, I agree.
On Sat, Mar 15, 2014 at 2:38 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Mar 14, 2014 at 12:06:21PM -0400, Josh Boyer wrote:
I think the first is preferable; the main case where you get a kernel without the drivers package is when you're building something intended to be small, and going downwards isn't the way to really do that.
That protection needs to be done in yum/dnf then. I don't see a way to do it cleanly in the kernel packages themselves.
Yeah, I agree.
And so would I. But it seems the dnf folks made this decision very clear several times over a longer time span (like closing a bug in June 2013, stating the fact in January 2014 and it's even been written down in the docs) so it's unlikely to be changed. And since dnf is planned for F22, changing yum's behavior would be much work for only just one release. Also, as Josh stated, removing kernel-drivers isn't that critical. Therefore, I vote we don't touch yum.
-- Sandro
On Wed, Mar 05, 2014 at 10:02:17AM -0500, Josh Boyer wrote:
On Wed, Mar 5, 2014 at 9:54 AM, Don Zickus dzickus@redhat.com wrote:
On Wed, Mar 05, 2014 at 08:25:12PM +0900, Sandro "red" Mathys wrote: For example, lets start with 100MB package requirement for the kernel (and say 2 GB for userspace). This way the kernel team can implement reasonable changes and monitor proper usage (because things grow over time).
If later on you realize 100 MB is not competitive enough, come back and chop it down to say 50 MB and let the kernel team figure it out.
But please do not come in here with a 'every MB counts' approach. It is not very sustainable for future growth nor really easy to implement from an engineering approach.
Is that acceptable? The kernel team can start with a hard limit of 100MB package requirement (or something reasonably less)? Let's work off budget requirements please.
This is a fair point. To be honest, I've ignored the "every MB counts" aspect entirely for now. I've instead been focusing on required functionality, because that's likely going to be the main driver of what the resulting size will be.
FWIW, the existing kernel package installed today (a debug kernel even) is ~142 MB. 123MB of that is the /lib/modules content. ~6MB of that is vmlinuz. The remaining 13MB is the initramfs, which is actually something that composes on the system during install and not something we can shrink from a packaging standpoint.
Also, I just arbitrarly threw out 100MB, if we should start higher, say 150MB, then it doesn't matter to me. :-)
Cheers, Don
On Wed, 5 Mar 2014 10:16:21 -0500 Don Zickus dzickus@redhat.com wrote:
Also, I just arbitrarly threw out 100MB, if we should start higher, say 150MB, then it doesn't matter to me. :-)
This entire disk size optimization seems kind of weird to me.
I just booted a f20 offiical cloud image in our openstack cloud. I used the m1.tiny (smallest size, no persistent storage):
Filesystem Size Used Avail Use% Mounted on /dev/vda1 20G 1.1G 19G 6% /
Is a few 10's of MB's really worth making our kernel a bunch more complex? Is disk space the right thing to be trying to optimize?
Perhaps I am missing it, but are there cases where the current cloud image is too large? what are they?
kevin
On Wed, Mar 05, 2014 at 09:28:45AM -0700, Kevin Fenzi wrote:
On Wed, 5 Mar 2014 10:16:21 -0500 Don Zickus dzickus@redhat.com wrote:
Also, I just arbitrarly threw out 100MB, if we should start higher, say 150MB, then it doesn't matter to me. :-)
This entire disk size optimization seems kind of weird to me.
I just booted a f20 offiical cloud image in our openstack cloud. I used the m1.tiny (smallest size, no persistent storage):
Filesystem Size Used Avail Use% Mounted on /dev/vda1 20G 1.1G 19G 6% /
Is a few 10's of MB's really worth making our kernel a bunch more complex? Is disk space the right thing to be trying to optimize?
Perhaps I am missing it, but are there cases where the current cloud image is too large? what are they?
Haha. Great point!
Cheers, Don
Am 05.03.2014 17:28, schrieb Kevin Fenzi:
On Wed, 5 Mar 2014 10:16:21 -0500 Don Zickus dzickus@redhat.com wrote:
Also, I just arbitrarly threw out 100MB, if we should start higher, say 150MB, then it doesn't matter to me. :-)
This entire disk size optimization seems kind of weird to me.
in case of kernel agreed in general you need to multiply the wasted space for each instance
I just booted a f20 offiical cloud image in our openstack cloud. I used the m1.tiny (smallest size, no persistent storage):
Filesystem Size Used Avail Use% Mounted on /dev/vda1 20G 1.1G 19G 6% /
way too fat and i even regret the 5.8 GB rootfs on 30 instances but was unsure in 2008 how much reservation would be needed
[root@proxy:~]$ df Filesystem Type Size Used Avail Use% Mounted on /dev/sdb1 ext4 5.8G 681M 5.1G 12% / /dev/sda1 ext4 493M 34M 460M 7% /boot
On Wed, 05 Mar 2014 17:37:42 +0100 Reindl Harald h.reindl@thelounge.net wrote:
Am 05.03.2014 17:28, schrieb Kevin Fenzi:
On Wed, 5 Mar 2014 10:16:21 -0500 Don Zickus dzickus@redhat.com wrote:
Also, I just arbitrarly threw out 100MB, if we should start higher, say 150MB, then it doesn't matter to me. :-)
This entire disk size optimization seems kind of weird to me.
in case of kernel agreed in general you need to multiply the wasted space for each instance
At least for my uses, the amount of non persistent disk space isn't a big deal. If I need disk space, I would attach a persistent volume...
I'd care much more about memory usage, boot time (you want to spin things up fast), and cpu usage (cpus are likely the most scarce resource for me at least).
kevin
Am 05.03.2014 17:45, schrieb Kevin Fenzi:
On Wed, 05 Mar 2014 17:37:42 +0100 Reindl Harald h.reindl@thelounge.net wrote:
Am 05.03.2014 17:28, schrieb Kevin Fenzi:
On Wed, 5 Mar 2014 10:16:21 -0500 Don Zickus dzickus@redhat.com wrote:
Also, I just arbitrarly threw out 100MB, if we should start higher, say 150MB, then it doesn't matter to me. :-)
This entire disk size optimization seems kind of weird to me.
in case of kernel agreed in general you need to multiply the wasted space for each instance
At least for my uses, the amount of non persistent disk space isn't a big deal. If I need disk space, I would attach a persistent volume...
in my use cases the SAN storage has already 12 SAS 15k disks and the additional enclosure as well as the disks are expensive
well, i have rotating backups of the whole setups too and so at least i have to multiply the storage by 3 or 4
I'd care much more about memory usage, boot time (you want to spin things up fast), and cpu usage (cpus are likely the most scarce resource for me at least)
30 Fedora VM's on a single HP DL 380 average 5% CPU usage
that different the usecases are
On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi kevin@scrye.com wrote:
On Wed, 05 Mar 2014 17:37:42 +0100 Reindl Harald h.reindl@thelounge.net wrote:
in general you need to multiply the wasted space for each instance
Exactly, you usually have hundreds or even thousands of instances running. Sure, "every MB counts" isn't to be taken literal here, maybe I should rather have said "every 10 MB count".
At least for my uses, the amount of non persistent disk space isn't a big deal. If I need disk space, I would attach a persistent volume...
Figure you get your additional persistent volumes for free somehow, so all those Amazon AWS, HP Cloud, Rackspace, etc. users envy you. And those admins that need to buy physical disks to put into their private clouds, too.
Also, more data equals more network traffic and more time - both things that matter in terms of costs, at least in public clouds.
-- Sandro
On Thu, Mar 6, 2014 at 12:38 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi kevin@scrye.com wrote:
On Wed, 05 Mar 2014 17:37:42 +0100 Reindl Harald h.reindl@thelounge.net wrote:
in general you need to multiply the wasted space for each instance
Exactly, you usually have hundreds or even thousands of instances running. Sure, "every MB counts" isn't to be taken literal here, maybe I should rather have said "every 10 MB count".
I suggested this once before and got no real answer but if you are so disk space constrained wouldn't file system compression do what you want instead of trying to micoroptimize every package?
Am 06.03.2014 16:00, schrieb drago01:
On Thu, Mar 6, 2014 at 12:38 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi kevin@scrye.com wrote:
On Wed, 05 Mar 2014 17:37:42 +0100 Reindl Harald h.reindl@thelounge.net wrote:
in general you need to multiply the wasted space for each instance
Exactly, you usually have hundreds or even thousands of instances running. Sure, "every MB counts" isn't to be taken literal here, maybe I should rather have said "every 10 MB count".
I suggested this once before and got no real answer but if you are so disk space constrained wouldn't file system compression do what you want instead of trying to micoroptimize every package?
since there is no transparent compression for the rootfs and even after btrfs get stable you shouldn't just convert existing installations from 2008 that's no option
reducing package sizes would affect 5 years old setups too but as said - i see more sane potential to split the glibc locales out to sub packages which are as big as the kernel then risk breaking updates because needed modules are in a now extra package
another thing would be split docs and manpages in general in subpackages - only god knows if and when DNF supports "tsflags=nodocs" which is driven by a yum plugin
another thing would be to split database support of libraries in subpackages -> dbmail pulls libzdb which pulls mysql, postgresql.. and take a deeper look in circular dependencies at all, i had several "nice to have" things on my servers and removing some of them ended in a freed dependency chain uninstalling 400 MB on each machine and even more on some - over the years i reduced my rootfs-usage down to 40-50% that way
some of these expensive dependencies i solved by maintain such packages at my own and disable postgresql support completly - but that's only a sane way in a defined environment
so again: there are *many* places to gain more than with the kernel itself and especially in case of C software yum pulls such subpackages in most cases by implicit dependencies
On Fri, Mar 7, 2014 at 12:00 AM, drago01 drago01@gmail.com wrote:
On Thu, Mar 6, 2014 at 12:38 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi kevin@scrye.com wrote:
On Wed, 05 Mar 2014 17:37:42 +0100 Reindl Harald h.reindl@thelounge.net wrote:
in general you need to multiply the wasted space for each instance
Exactly, you usually have hundreds or even thousands of instances running. Sure, "every MB counts" isn't to be taken literal here, maybe I should rather have said "every 10 MB count".
I suggested this once before and got no real answer but if you are so disk space constrained wouldn't file system compression do what you want instead of trying to micoroptimize every package?
Because file system compression requires cpu cycles (i.e. cpu power and time) and I think they are more valuable to most users than storage. So, not optimizing the packages would probably be preferable over compressing the file system. But trimmed down packages need less space but no additional cpu cycles so it's a win-win from the user's perspective.
-- Sandro
On Thu, Mar 06, 2014 at 08:38:44AM +0900, Sandro "red" Mathys wrote:
On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi kevin@scrye.com wrote:
On Wed, 05 Mar 2014 17:37:42 +0100 Reindl Harald h.reindl@thelounge.net wrote:
in general you need to multiply the wasted space for each instance
Exactly, you usually have hundreds or even thousands of instances running. Sure, "every MB counts" isn't to be taken literal here, maybe I should rather have said "every 10 MB count".
At least for my uses, the amount of non persistent disk space isn't a big deal. If I need disk space, I would attach a persistent volume...
Figure you get your additional persistent volumes for free somehow, so all those Amazon AWS, HP Cloud, Rackspace, etc. users envy you. And those admins that need to buy physical disks to put into their private clouds, too.
Also, more data equals more network traffic and more time - both things that matter in terms of costs, at least in public clouds.
Sure, but what if the trade-off in size comes with a cost in speed? Is cloud ok with the kernel taking twice as long to boot? Or maybe running slower? Or maybe crashing more often (because we removed safety checks?).
I mean if Josh wanted to he could make everything modular and have a really small kernel footprint (like 40MB or so) running in 50MB of memory (I have done this with kdump). But it costs you speed in loading modules (as opposed to built into the kernel). You may lose other optional optimizations that help speed things up.
Other SIGs made not like it, but again it depends on how you frame your environment. Maybe cloud really needs its own kernel. We don't know.
What is cloud willing to sacrafice to obtain smaller size?
Cheers, Don
On Thu, Mar 06, 2014 at 10:04:56AM -0500, Don Zickus wrote:
On Thu, Mar 06, 2014 at 08:38:44AM +0900, Sandro "red" Mathys wrote:
On Thu, Mar 6, 2014 at 1:45 AM, Kevin Fenzi kevin@scrye.com wrote:
On Wed, 05 Mar 2014 17:37:42 +0100 Reindl Harald h.reindl@thelounge.net wrote:
in general you need to multiply the wasted space for each instance
Exactly, you usually have hundreds or even thousands of instances running. Sure, "every MB counts" isn't to be taken literal here, maybe I should rather have said "every 10 MB count".
At least for my uses, the amount of non persistent disk space isn't a big deal. If I need disk space, I would attach a persistent volume...
Figure you get your additional persistent volumes for free somehow, so all those Amazon AWS, HP Cloud, Rackspace, etc. users envy you. And those admins that need to buy physical disks to put into their private clouds, too.
Also, more data equals more network traffic and more time - both things that matter in terms of costs, at least in public clouds.
Sure, but what if the trade-off in size comes with a cost in speed? Is cloud ok with the kernel taking twice as long to boot? Or maybe running slower? Or maybe crashing more often (because we removed safety checks?).
I mean if Josh wanted to he could make everything modular and have a really small kernel footprint (like 40MB or so) running in 50MB of memory (I have done this with kdump). But it costs you speed in loading modules (as opposed to built into the kernel). You may lose other optional optimizations that help speed things up.
Other SIGs made not like it, but again it depends on how you frame your environment. Maybe cloud really needs its own kernel. We don't know.
To touch on this a bit, if splitting the kernel up becomes unwieldy we could possibly just build kernel-cloud as another flavour in kernel.spec (similar to PAE, debug, etc). I'm not much of a fan of this though, as it can lead to a very divergent vmlinux over time.
If it's _necessary_, that's one thing. I've yet to really see any data backing up necessity on any of this at all though. Right now it seems to be sitting in the "nice to have" category.
Perhaps someone from the cloud team could look at existing images from other distros and figure out kernel sizes there, and how it plays into usage and cost in those environments?
josh
On Thu, Mar 06, 2014 at 11:02:47AM -0500, Josh Boyer wrote:
If it's _necessary_, that's one thing. I've yet to really see any data backing up necessity on any of this at all though. Right now it seems to be sitting in the "nice to have" category.
For the record, it is _literally_ sitting in our "nice to have" category. See https://fedoraproject.org/wiki/Cloud_Changelist#Change:_Cloud-Friendly_Kerne...
:)
Perhaps someone from the cloud team could look at existing images from other distros and figure out kernel sizes there, and how it plays into usage and cost in those environments?
On the ubuntu EC2 image, /lib/modules/$(uname -r) is 24M + 5.2M vmlinuz + 1.1M in /lib/firmware. Total package size is 32M on disk. And 5.9M initrd.
CoreOS is bigger, with 33M in /lib/modules and 5.2M in lib/firmware, and a /19M vmlinuz.
Which may just go to show that _calling_ yourself ultra-minimal and focused is actually more important than _being_ that.
On Thu, Mar 06, 2014 at 11:32:55AM -0500, Matthew Miller wrote:
On Thu, Mar 06, 2014 at 11:02:47AM -0500, Josh Boyer wrote:
If it's _necessary_, that's one thing. I've yet to really see any data backing up necessity on any of this at all though. Right now it seems to be sitting in the "nice to have" category.
For the record, it is _literally_ sitting in our "nice to have" category. See https://fedoraproject.org/wiki/Cloud_Changelist#Change:_Cloud-Friendly_Kerne...
:)
Perhaps someone from the cloud team could look at existing images from other distros and figure out kernel sizes there, and how it plays into usage and cost in those environments?
On the ubuntu EC2 image, /lib/modules/$(uname -r) is 24M + 5.2M vmlinuz + 1.1M in /lib/firmware. Total package size is 32M on disk. And 5.9M initrd.
CoreOS is bigger, with 33M in /lib/modules and 5.2M in lib/firmware, and a /19M vmlinuz.
Yeah, hard numbers to compete with! :-)
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
He just has to cut 35MB or so from /lib/modules/. We can probably nickel and dime and review a lot of cruft to get there, but what is that 35MB really doing to get us anything? I am sure half of that can be removed by re-examining the minimal-list he sent (I can even help there).
Maybe impose only xfs as the fs of choice or some other restrictions and chop it further, but then we lose flexibility.
Instead of competing with Ubuntu on minimalist can we compete on pretty close but a lot more flexible? Do Ubuntu users have much choice on how they configure their environment? Or is Fedora Cloud providing a generic cookie cutter installation?
Cheers, Don
Am 06.03.2014 18:04, schrieb Don Zickus:
Maybe impose only xfs as the fs of choice or some other restrictions and chop it further, but then we lose flexibility
hopefully a joke :-)
* i use a ton of virtual Fedora instances * any of them is using ext4 * some may benefit from XFS but not that much to have different base environments and spread the test matrix for updates * even if BTRFS is the overall default in a few years most of them are installed 2008 and dist-upgraded over the years, tiny, clean and perfectly maintained * i do not plan a from-scratch install as long as i live
filesystems may hahve their pros and cons but as admin the most benefit is limit the things you are using and know how to handle them in trouble, 1% performance is nice but the price of a mistake because things are working not as usual in border cases is way too high
disclaimer: maybe not perfect english but i think the point is clear
On Thu, Mar 6, 2014 at 12:04 PM, Don Zickus dzickus@redhat.com wrote:
On Thu, Mar 06, 2014 at 11:32:55AM -0500, Matthew Miller wrote:
On Thu, Mar 06, 2014 at 11:02:47AM -0500, Josh Boyer wrote:
If it's _necessary_, that's one thing. I've yet to really see any data backing up necessity on any of this at all though. Right now it seems to be sitting in the "nice to have" category.
For the record, it is _literally_ sitting in our "nice to have" category. See https://fedoraproject.org/wiki/Cloud_Changelist#Change:_Cloud-Friendly_Kerne...
:)
Perhaps someone from the cloud team could look at existing images from other distros and figure out kernel sizes there, and how it plays into usage and cost in those environments?
On the ubuntu EC2 image, /lib/modules/$(uname -r) is 24M + 5.2M vmlinuz + 1.1M in /lib/firmware. Total package size is 32M on disk. And 5.9M initrd.
CoreOS is bigger, with 33M in /lib/modules and 5.2M in lib/firmware, and a /19M vmlinuz.
Yeah, hard numbers to compete with! :-)
The only way to win is to not play at all? :)
Small note too, just because the vmlinuz are of a certain size does not mean they contain similar content. Without really digging into the config settings it's hard to do a true apples to apples comparison. Still, having the overall sizes handy is helpful, thanks.
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
He just has to cut 35MB or so from /lib/modules/. We can probably nickel and dime and review a lot of cruft to get there, but what is that 35MB really doing to get us anything? I am sure half of that can be removed by re-examining the minimal-list he sent (I can even help there).
Right. Considering the bloat elsewhere in the distro, I think we can start with what I have and work from there if needed.
Maybe impose only xfs as the fs of choice or some other restrictions and chop it further, but then we lose flexibility.
Oh dear. Please not another FS thread. So many emails from last week...
Instead of competing with Ubuntu on minimalist can we compete on pretty close but a lot more flexible? Do Ubuntu users have much choice on how they configure their environment? Or is Fedora Cloud providing a generic cookie cutter installation?
Right, I kind of like that we'd have a smaller core package that is still broadly useful.
josh
On Thu, Mar 06, 2014 at 01:10:46PM -0500, Josh Boyer wrote:
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
Currently, this is required by the kernel for pre but not at runtime. (I guess /bin/kernel-install ends up putting things into the initramfs.) Is there any easy way to do that differently and any win from it?
Instead of competing with Ubuntu on minimalist can we compete on pretty close but a lot more flexible? Do Ubuntu users have much choice on how they configure their environment? Or is Fedora Cloud providing a generic cookie cutter installation?
Right, I kind of like that we'd have a smaller core package that is still broadly useful.
+1
On Thu, Mar 6, 2014 at 1:43 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Thu, Mar 06, 2014 at 01:10:46PM -0500, Josh Boyer wrote:
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
Currently, this is required by the kernel for pre but not at runtime. (I guess /bin/kernel-install ends up putting things into the initramfs.) Is there any easy way to do that differently and any win from it?
Possibly. The majority of the firmware is needed by the drivers. If those drivers aren't in kernel-core, the pre isn't needed on kernel-core. It would remain on kernel and possibly be added to kernel-drivers.
Aside from that (or in combination with it), it would basically be splitting up linux-firmware into more subpackages or maybe putting requires on specific files.
josh
On Fri, Mar 7, 2014 at 3:10 AM, Josh Boyer jwboyer@fedoraproject.org wrote:
On Thu, Mar 6, 2014 at 12:04 PM, Don Zickus dzickus@redhat.com wrote:
On Thu, Mar 06, 2014 at 11:32:55AM -0500, Matthew Miller wrote:
On Thu, Mar 06, 2014 at 11:02:47AM -0500, Josh Boyer wrote:
If it's _necessary_, that's one thing. I've yet to really see any data backing up necessity on any of this at all though. Right now it seems to be sitting in the "nice to have" category.
For the record, it is _literally_ sitting in our "nice to have" category. See https://fedoraproject.org/wiki/Cloud_Changelist#Change:_Cloud-Friendly_Kerne...
:)
Perhaps someone from the cloud team could look at existing images from other distros and figure out kernel sizes there, and how it plays into usage and cost in those environments?
On the ubuntu EC2 image, /lib/modules/$(uname -r) is 24M + 5.2M vmlinuz + 1.1M in /lib/firmware. Total package size is 32M on disk. And 5.9M initrd.
CoreOS is bigger, with 33M in /lib/modules and 5.2M in lib/firmware, and a /19M vmlinuz.
Yeah, hard numbers to compete with! :-)
I really think comparing our cloud images to theirs is comparing apples to !apples. The numbers are nice to know what we're competing against, but don't think we need to have the smallest kernel.
I don't know this for a fact, but I think the Ubuntu EC2 image is tailored very specifically to run in EC2 only. We don't do this kind of tailoring in Fedora. One to rule them all.
And CoreOS is very tailored, and has different images for about everything they run on so it's fair to assume they also have several different kernel setups. We only want two: core/cloud and full.
The only way to win is to not play at all? :)
Small note too, just because the vmlinuz are of a certain size does not mean they contain similar content. Without really digging into the config settings it's hard to do a true apples to apples comparison. Still, having the overall sizes handy is helpful, thanks.
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
He just has to cut 35MB or so from /lib/modules/. We can probably nickel and dime and review a lot of cruft to get there, but what is that 35MB really doing to get us anything? I am sure half of that can be removed by re-examining the minimal-list he sent (I can even help there).
Right. Considering the bloat elsewhere in the distro, I think we can start with what I have and work from there if needed.
Excellent progress there. So 35MB are already gone and I figure (re)moving graphics, sound and other obvious things will gain us quite some more MB. Nice job.
Maybe impose only xfs as the fs of choice or some other restrictions and chop it further, but then we lose flexibility.
Oh dear. Please not another FS thread. So many emails from last week...
Instead of competing with Ubuntu on minimalist can we compete on pretty close but a lot more flexible? Do Ubuntu users have much choice on how they configure their environment? Or is Fedora Cloud providing a generic cookie cutter installation?
Right, I kind of like that we'd have a smaller core package that is still broadly useful.
Exactly. We only offer different images for different purposes (software stacks), not for different platforms. So every image should be able to run on all supported platforms.
-- Sandro
On Fri, Mar 7, 2014 at 2:16 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
He just has to cut 35MB or so from /lib/modules/. We can probably nickel and dime and review a lot of cruft to get there, but what is that 35MB really doing to get us anything? I am sure half of that can be removed by re-examining the minimal-list he sent (I can even help there).
Right. Considering the bloat elsewhere in the distro, I think we can start with what I have and work from there if needed.
Excellent progress there. So 35MB are already gone and I figure (re)moving graphics, sound and other obvious things will gain us quite some more MB. Nice job.
No. 62MB are already "gone". The 35MB was what is needed additionally to get to comparable numbers with ubuntu. Sound I could see dropping. Graphics, not so much.
Really, I'm likely to start with what I have and if there are major reasons that get brought up (with data) to trim further, we can look at it.
josh
On Fri, Mar 07, 2014 at 08:23:15AM -0500, Josh Boyer wrote:
On Fri, Mar 7, 2014 at 2:16 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
He just has to cut 35MB or so from /lib/modules/. We can probably nickel and dime and review a lot of cruft to get there, but what is that 35MB really doing to get us anything? I am sure half of that can be removed by re-examining the minimal-list he sent (I can even help there).
Right. Considering the bloat elsewhere in the distro, I think we can start with what I have and work from there if needed.
Excellent progress there. So 35MB are already gone and I figure (re)moving graphics, sound and other obvious things will gain us quite some more MB. Nice job.
No. 62MB are already "gone". The 35MB was what is needed additionally to get to comparable numbers with ubuntu. Sound I could see dropping. Graphics, not so much.
Really, I'm likely to start with what I have and if there are major reasons that get brought up (with data) to trim further, we can look at it.
I agree too. I think you did an awesome job Josh. Most of the fat is trimmed. :-) Anything more should really come with some numbers/data.
Cheers, Don
On Fri, Mar 7, 2014 at 8:59 AM, Don Zickus dzickus@redhat.com wrote:
On Fri, Mar 07, 2014 at 08:23:15AM -0500, Josh Boyer wrote:
On Fri, Mar 7, 2014 at 2:16 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
He just has to cut 35MB or so from /lib/modules/. We can probably nickel and dime and review a lot of cruft to get there, but what is that 35MB really doing to get us anything? I am sure half of that can be removed by re-examining the minimal-list he sent (I can even help there).
Right. Considering the bloat elsewhere in the distro, I think we can start with what I have and work from there if needed.
Excellent progress there. So 35MB are already gone and I figure (re)moving graphics, sound and other obvious things will gain us quite some more MB. Nice job.
No. 62MB are already "gone". The 35MB was what is needed additionally to get to comparable numbers with ubuntu. Sound I could see dropping. Graphics, not so much.
Really, I'm likely to start with what I have and if there are major reasons that get brought up (with data) to trim further, we can look at it.
I agree too. I think you did an awesome job Josh. Most of the fat is trimmed. :-) Anything more should really come with some numbers/data.
Don't get too heavy on the praise yet. This still needs to be done at the packaging level. The changes to kernel.spec might make you want to poke your eyes out or something ;).
josh
On Fri, Mar 07, 2014 at 09:50:37AM -0500, Josh Boyer wrote:
On Fri, Mar 7, 2014 at 8:59 AM, Don Zickus dzickus@redhat.com wrote:
On Fri, Mar 07, 2014 at 08:23:15AM -0500, Josh Boyer wrote:
On Fri, Mar 7, 2014 at 2:16 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
He just has to cut 35MB or so from /lib/modules/. We can probably nickel and dime and review a lot of cruft to get there, but what is that 35MB really doing to get us anything? I am sure half of that can be removed by re-examining the minimal-list he sent (I can even help there).
Right. Considering the bloat elsewhere in the distro, I think we can start with what I have and work from there if needed.
Excellent progress there. So 35MB are already gone and I figure (re)moving graphics, sound and other obvious things will gain us quite some more MB. Nice job.
No. 62MB are already "gone". The 35MB was what is needed additionally to get to comparable numbers with ubuntu. Sound I could see dropping. Graphics, not so much.
Really, I'm likely to start with what I have and if there are major reasons that get brought up (with data) to trim further, we can look at it.
I agree too. I think you did an awesome job Josh. Most of the fat is trimmed. :-) Anything more should really come with some numbers/data.
Don't get too heavy on the praise yet. This still needs to be done at the packaging level. The changes to kernel.spec might make you want to poke your eyes out or something ;).
Oh, I thought you had the package split already into kernel-common and kernel-drivers?
But yes I would be interesting in seeing those changes because that filters to RHEL eventually. I can already see RHEL developers whining about extra packages to install... ;-) Though all the kernel-tools stuff seemed to have gone smoothly...
Cheers, Don
On Fri, Mar 7, 2014 at 10:06 AM, Don Zickus dzickus@redhat.com wrote:
On Fri, Mar 07, 2014 at 09:50:37AM -0500, Josh Boyer wrote:
On Fri, Mar 7, 2014 at 8:59 AM, Don Zickus dzickus@redhat.com wrote:
On Fri, Mar 07, 2014 at 08:23:15AM -0500, Josh Boyer wrote:
On Fri, Mar 7, 2014 at 2:16 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
> I think Josh is mostly there. He has 58MB + 5M vmlinuz + <similar?> > firmwre.
Firmware is owned by linux-firmware, not the kernel package. I didn't include it in my kernel numbers for that reason.
> He just has to cut 35MB or so from /lib/modules/. We can probably nickel > and dime and review a lot of cruft to get there, but what is that 35MB > really doing to get us anything? I am sure half of that can be removed by > re-examining the minimal-list he sent (I can even help there).
Right. Considering the bloat elsewhere in the distro, I think we can start with what I have and work from there if needed.
Excellent progress there. So 35MB are already gone and I figure (re)moving graphics, sound and other obvious things will gain us quite some more MB. Nice job.
No. 62MB are already "gone". The 35MB was what is needed additionally to get to comparable numbers with ubuntu. Sound I could see dropping. Graphics, not so much.
Really, I'm likely to start with what I have and if there are major reasons that get brought up (with data) to trim further, we can look at it.
I agree too. I think you did an awesome job Josh. Most of the fat is trimmed. :-) Anything more should really come with some numbers/data.
Don't get too heavy on the praise yet. This still needs to be done at the packaging level. The changes to kernel.spec might make you want to poke your eyes out or something ;).
Oh, I thought you had the package split already into kernel-common and kernel-drivers?
I have the basics to create those packages and the kernel metapackage done. Right now they contain strictly /boot for kernel-core and /lib/modules for kernel-drivers. What I don't have done is the swizzling around of certain modules to be in kernel-core yet. The numbers gathering was just hacking a local VM on the filesystem directly.
So the next step is to translate that to the proper %files entries and build/test.
But yes I would be interesting in seeing those changes because that filters to RHEL eventually. I can already see RHEL developers whining about extra packages to install... ;-) Though all the kernel-tools stuff seemed to have gone smoothly...
Right. As soon as I get something that seems like it boots in a VM when you install it, I'll post the changes here for review. The current stuff I have done is in a COPR if you are _really_ itching to see it:
http://copr-fe.cloud.fedoraproject.org/coprs/jwboyer/kernel-core/
DO NOT INSTALL THAT. It's stale, won't boot, hasn't been tested, etc. I'll be building there for now as I work through things, but until further notice it only exists to make sure things build.
josh
On Wed, Mar 5, 2014 at 2:48 AM, Sandro "red" Mathys red@fedoraproject.org wrote:
So, in our case hardware drivers are rather unnecessary and the kernel experts might know other ways to shrink the footprint for our limited use cases. The kernel we require supports all primary architectures (i686 and x86_64 right now, ARM likely later) and is able to run on a hypervisor (primarily KVM and Xen but support for ESXi and Hyper-V are becoming crucial in private clouds). Only. No bare-metal. We also require support for Linux Containers (LXC) to enable the use of Docker.
Now, I heard some people screaming when I said HW drivers are unnecessary. Some people will want to make use of PCI passthrough and while I think we don't want to ship the necessary modules by default, they should be easily installable through a separate package. If some more granularity is acceptable (and makes sense), I think one package for SRIOV NICs, one for graphic cards (to enable mathematical computation stuff) and one for everything else PCI passthrough (plus one for all other HW drivers for the non-cloud products) would be totally nice.
I'm not overly thrilled with having multi-tiered driver packages. That leads to major headaches when we start shuffling things around from one driver package to another. The current solution I have prototyped is a kernel-core/kernel-drivers split. Here "core" could be analogous to "cloud", but I imagine it will be useful for other things. People wanting to do PCI passthrough can just install the kernel-drivers package.
What does everybody think about this? Can it be done? How is it best done? What's the timeframe (we'd really like to see this implemented in F21 Beta but obviously the earlier modular kernels can be tested, the better)? Do you require additional input?
Same questions I asked Sam and Matt:
- Do you need/want a firewall (requires iptables, etc)? - Do you need/want NFS or other cloudy storage things (for gluster?)? - Do you need/want openvswitch?
The list of modules I have in my local rawhide KVM guest is below. The snd_* related drivers probably aren't necessary. The btrfs and things it depends on can be ignored (unless you plan on switching from ext4). Anything that has table or nf in the module name is for the firewall.
Matt already provided a much smaller module list for openstack and EC2, but I'm guessing we want to target the broadest usecase. Think about it and let me know.
josh
[jwboyer@localhost ~]$ lsmod Module Size Used by nls_utf8 12557 1 isofs 39794 1 uinput 17708 1 bnep 19735 2 bluetooth 445507 5 bnep 6lowpan_iphc 18591 1 bluetooth fuse 91190 3 ip6t_rpfilter 12546 1 ip6t_REJECT 12939 2 xt_conntrack 12760 9 cfg80211 583354 0 rfkill 22195 4 cfg80211,bluetooth ebtable_nat 12807 0 ebtable_broute 12731 0 bridge 135391 1 ebtable_broute stp 12946 1 bridge llc 14092 2 stp,bridge ebtable_filter 12827 0 ebtables 30833 3 ebtable_broute,ebtable_nat,ebtable_filter ip6table_nat 13015 1 nf_conntrack_ipv6 18777 6 nf_defrag_ipv6 100248 1 nf_conntrack_ipv6 nf_nat_ipv6 13213 1 ip6table_nat ip6table_mangle 12700 1 ip6table_security 12710 1 ip6table_raw 12683 1 ip6table_filter 12815 1 ip6_tables 26809 5 ip6table_filter,ip6table_mangle,ip6table_security,ip6table_nat,ip6table_raw iptable_nat 13011 1 nf_conntrack_ipv4 18791 5 nf_defrag_ipv4 12702 1 nf_conntrack_ipv4 nf_nat_ipv4 13199 1 iptable_nat nf_nat 25249 4 nf_nat_ipv4,nf_nat_ipv6,ip6table_nat,iptable_nat nf_conntrack 110550 8 nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,ip6table_nat,iptable_nat,nf_conntrack_ipv4,nf_conntrack_ipv6 iptable_mangle 12695 1 iptable_security 12705 1 iptable_raw 12678 1 snd_hda_codec_generic 66943 1 ppdev 17635 0 snd_hda_intel 56588 4 snd_hda_codec 133858 2 snd_hda_codec_generic,snd_hda_intel snd_hwdep 17650 1 snd_hda_codec snd_seq 65180 0 snd_seq_device 14136 1 snd_seq crct10dif_pclmul 14250 0 crc32_pclmul 13113 0 crc32c_intel 22079 0 snd_pcm 103502 2 snd_hda_codec,snd_hda_intel ghash_clmulni_intel 13259 0 microcode 216608 0 virtio_console 28109 1 serio_raw 13413 0 snd_timer 28806 2 snd_pcm,snd_seq virtio_balloon 13530 0 snd 83790 16 snd_hwdep,snd_timer,snd_pcm,snd_seq,snd_hda_codec_generic,snd_hda_codec,snd_hda_intel,snd_seq_device soundcore 14491 1 snd parport_pc 28048 0 parport 40605 2 ppdev,parport_pc pvpanic 12801 0 i2c_piix4 22155 0 btrfs 940276 1 xor 21366 1 btrfs raid6_pq 101472 1 btrfs qxl 74078 2 drm_kms_helper 50413 1 qxl virtio_pci 17713 0 8139too 33711 0 virtio_ring 20004 3 virtio_pci,virtio_balloon,virtio_console virtio 14172 3 virtio_pci,virtio_balloon,virtio_console ttm 85373 1 qxl 8139cp 32036 0 mii 13527 2 8139cp,8139too ata_generic 12910 0 drm 288814 4 qxl,ttm,drm_kms_helper i2c_core 38734 3 drm,i2c_piix4,drm_kms_helper pata_acpi 13038 0
Hi,
I'm not overly thrilled with having multi-tiered driver packages. That leads to major headaches when we start shuffling things around from one driver package to another. The current solution I have prototyped is a kernel-core/kernel-drivers split. Here "core" could be analogous to "cloud", but I imagine it will be useful for other things. People wanting to do PCI passthrough can just install the kernel-drivers package.
Agree, I don't think it makes much sense to split things into many small pieces.
- Do you need/want a firewall (requires iptables, etc)?
I'd say yes by default, but being able to remove it might be useful (kernel-netfilter subpackage)?
- Do you need/want NFS or other cloudy storage things (for gluster?)?
I'm personally using NFS for host/guest filesystem sharing, so I'd vote yes.
- Do you need/want openvswitch?
I think no. That's a host side thing and this is about the guest kernel, right?
The list of modules I have in my local rawhide KVM guest is below. The snd_* related drivers probably aren't necessary. The btrfs and things it depends on can be ignored (unless you plan on switching from ext4).
Depends on what is targeted. Strictly cloud? Or also someone running Fedora as a guest in virt-manager / boxes?
I'd tend to include drivers for pretty much everything qemu can emulate.
uinput 17708 1
spice guest agent uses this.
bluetooth 445507 5 bnep cfg80211 583354 0
bluetooth+wireless are not needed, dunno why they are loaded.
snd_hda_codec_generic 66943 1 snd_hda_intel 56588 4
For the qemu HDA soundcard. Should be included if desktop usecase should be covered too.
qxl 74078 2
gpu driver. There are also cirrus + bochs-drm for qemu. vmware vga has a drm driver too. They should be included.
ata_generic 12910 0 pata_acpi 13038 0
IDE. Needed. Qemu can emulate AHCI too, should be included (but I think that is =y anyway).
Qemu can emulate a bunch of SCSI HBAs. Don't think we need the drivers for them (except virtio-scsi). vmware emulates SCSI HBAs too. pvscsi should be included. Dunno about the older ones, there was for example some buslogic emulation in older vmware workstation versions ...
USB: Qemu can also emulate uhci + ehci + xhci, should be included. Also usb-storage and generic hid support (for the usb tablet). IIRC most of this is =y anyway.
NIC: e1000, rtl8139 (qemu), tulip (hyperv), dunno about vmware. qemu can also emulate a bunch of other nics such as ne2k, but I think those are not relevant in practice and not needed.
Of course all paravirtual drivers should be included too, i.e. * all virtio (for qemu) * all xen frontend (for xen, the backend drivers are for the host side and are not needed in the guest). * all hyperv (for hyperv)
HTH, Gerd
On Wed, Mar 5, 2014 at 9:59 AM, Gerd Hoffmann kraxel@redhat.com wrote:
Hi,
I'm not overly thrilled with having multi-tiered driver packages. That leads to major headaches when we start shuffling things around from one driver package to another. The current solution I have prototyped is a kernel-core/kernel-drivers split. Here "core" could be analogous to "cloud", but I imagine it will be useful for other things. People wanting to do PCI passthrough can just install the kernel-drivers package.
Agree, I don't think it makes much sense to split things into many small pieces.
- Do you need/want a firewall (requires iptables, etc)?
I'd say yes by default, but being able to remove it might be useful (kernel-netfilter subpackage)?
So you agree multi-tiered subpackages is a bad idea, but then you propose a netfilter specific subpackage? ... Probably not. They'll likely just be in kernel-core.
- Do you need/want NFS or other cloudy storage things (for gluster?)?
I'm personally using NFS for host/guest filesystem sharing, so I'd vote yes.
- Do you need/want openvswitch?
I think no. That's a host side thing and this is about the guest kernel, right?
Nested virt? I dunno, it was mentioned at one point so I thought I'd bring it up for discussion again.
The list of modules I have in my local rawhide KVM guest is below. The snd_* related drivers probably aren't necessary. The btrfs and things it depends on can be ignored (unless you plan on switching from ext4).
Depends on what is targeted. Strictly cloud? Or also someone running Fedora as a guest in virt-manager / boxes?
I'm of the opinion the latter is probably what we should shoot for. It's going to be the broadest target that is still reasonably small.
I'd tend to include drivers for pretty much everything qemu can emulate.
uinput 17708 1
spice guest agent uses this.
bluetooth 445507 5 bnep cfg80211 583354 0
bluetooth+wireless are not needed, dunno why they are loaded.
Me either. I'm guessing because systemd/NetworkManager on my KVM guest auto-loads them.
snd_hda_codec_generic 66943 1 snd_hda_intel 56588 4
For the qemu HDA soundcard. Should be included if desktop usecase should be covered too.
qxl 74078 2
gpu driver. There are also cirrus + bochs-drm for qemu. vmware vga has a drm driver too. They should be included.
ata_generic 12910 0 pata_acpi 13038 0
IDE. Needed. Qemu can emulate AHCI too, should be included (but I think that is =y anyway).
Qemu can emulate a bunch of SCSI HBAs. Don't think we need the drivers for them (except virtio-scsi). vmware emulates SCSI HBAs too. pvscsi should be included. Dunno about the older ones, there was for example some buslogic emulation in older vmware workstation versions ...
USB: Qemu can also emulate uhci + ehci + xhci, should be included. Also usb-storage and generic hid support (for the usb tablet). IIRC most of this is =y anyway.
NIC: e1000, rtl8139 (qemu), tulip (hyperv), dunno about vmware. qemu can also emulate a bunch of other nics such as ne2k, but I think those are not relevant in practice and not needed.
Of course all paravirtual drivers should be included too, i.e.
- all virtio (for qemu)
- all xen frontend (for xen, the backend drivers are for the host side and are not needed in the guest).
- all hyperv (for hyperv)
OK. Was planning on that already.
josh
Hi,
Agree, I don't think it makes much sense to split things into many small pieces.
- Do you need/want a firewall (requires iptables, etc)?
I'd say yes by default, but being able to remove it might be useful (kernel-netfilter subpackage)?
So you agree multi-tiered subpackages is a bad idea,
On the hardware support side it is a bad idea I think. Too many different use cases and lines between them are blurry, so I suspect it becomes messy quickly if you try to split it fine-grained.
but then you propose a netfilter specific subpackage? ... Probably not. They'll likely just be in kernel-core.
"all netfilter modules" is pretty clear and so is the use case (=want firewall). But maybe not worth the trouble, didn't check what size they sum up to.
cheers, Gerd
On Wed, Mar 05, 2014 at 10:08:04 -0500, Josh Boyer jwboyer@fedoraproject.org wrote:
So you agree multi-tiered subpackages is a bad idea, but then you propose a netfilter specific subpackage? ... Probably not. They'll likely just be in kernel-core.
Couldn't the planned module provides allow something like this to work without worry much about which modules are in which packages? netfilter could juts require the modules it needs and it would pull in and extra kernel module packages that are needed.
On Wed, Mar 5, 2014 at 10:51 AM, Bruno Wolff III bruno@wolff.to wrote:
On Wed, Mar 05, 2014 at 10:08:04 -0500, Josh Boyer jwboyer@fedoraproject.org wrote:
So you agree multi-tiered subpackages is a bad idea, but then you propose a netfilter specific subpackage? ... Probably not. They'll likely just be in kernel-core.
Couldn't the planned module provides allow something like this to work without worry much about which modules are in which packages? netfilter could juts require the modules it needs and it would pull in and extra kernel module packages that are needed.
Possibly. But it's not just the installed system to be taken into account. Every additional subpackage and list is something that needs to be maintained and further complicates the kernel.spec file. It adds to testing all the combinations of kernel packages and subpackages being installed (or not) and upgraded, etc. For the sake of the kernel maintainers, keeping it limited is probably a good idea.
And honestly, a firewall in today's world seems pretty darn "core" to me anyway.
josh
On Wed, 2014-03-05 at 10:59 -0500, Josh Boyer wrote:
On Wed, Mar 5, 2014 at 10:51 AM, Bruno Wolff III bruno@wolff.to wrote:
On Wed, Mar 05, 2014 at 10:08:04 -0500, Josh Boyer jwboyer@fedoraproject.org wrote:
So you agree multi-tiered subpackages is a bad idea, but then you propose a netfilter specific subpackage? ... Probably not. They'll likely just be in kernel-core.
Couldn't the planned module provides allow something like this to work without worry much about which modules are in which packages? netfilter could juts require the modules it needs and it would pull in and extra kernel module packages that are needed.
Possibly. But it's not just the installed system to be taken into account. Every additional subpackage and list is something that needs to be maintained and further complicates the kernel.spec file. It adds to testing all the combinations of kernel packages and subpackages being installed (or not) and upgraded, etc. For the sake of the kernel maintainers, keeping it limited is probably a good idea.
This can also have unforseen consequences. Coming from conary, where we had rich dependencies stored in the repository, if you start putting module provides and depends metadata into the repositories for every single module, the CPU time and network traffic required to even do a simple update can grow excessively. I am not saying that there is no solution to the problem, but I wouldn't count on it near term.
Justin
On Wed, Mar 05, 2014 at 11:33:57AM -0600, Justin M. Forbes wrote:
On Wed, 2014-03-05 at 10:59 -0500, Josh Boyer wrote:
On Wed, Mar 5, 2014 at 10:51 AM, Bruno Wolff III bruno@wolff.to wrote:
On Wed, Mar 05, 2014 at 10:08:04 -0500, Josh Boyer jwboyer@fedoraproject.org wrote:
So you agree multi-tiered subpackages is a bad idea, but then you propose a netfilter specific subpackage? ... Probably not. They'll likely just be in kernel-core.
Couldn't the planned module provides allow something like this to work without worry much about which modules are in which packages? netfilter could juts require the modules it needs and it would pull in and extra kernel module packages that are needed.
Possibly. But it's not just the installed system to be taken into account. Every additional subpackage and list is something that needs to be maintained and further complicates the kernel.spec file. It adds to testing all the combinations of kernel packages and subpackages being installed (or not) and upgraded, etc. For the sake of the kernel maintainers, keeping it limited is probably a good idea.
This can also have unforseen consequences. Coming from conary, where we had rich dependencies stored in the repository, if you start putting module provides and depends metadata into the repositories for every single module, the CPU time and network traffic required to even do a simple update can grow excessively. I am not saying that there is no solution to the problem, but I wouldn't count on it near term.
Yeah, this is the biggest downside to auto-generated Provides for modules. I haven't had time to flip the switch and see how much it impacts things yet. If it results in any form of noticable impact we need to think long and hard about whether it's actually worth doing.
josh
On Wed, Mar 05, 2014 at 10:08:04AM -0500, Josh Boyer wrote:
Depends on what is targeted. Strictly cloud? Or also someone running Fedora as a guest in virt-manager / boxes?
I'm of the opinion the latter is probably what we should shoot for. It's going to be the broadest target that is still reasonably small.
I agree that this is valuable, because it lets the same image be used for testing locally.
On Thu, Mar 6, 2014 at 12:08 AM, Josh Boyer jwboyer@fedoraproject.org wrote:
On Wed, Mar 5, 2014 at 9:59 AM, Gerd Hoffmann kraxel@redhat.com wrote:
Hi,
I'm not overly thrilled with having multi-tiered driver packages. That leads to major headaches when we start shuffling things around from one driver package to another. The current solution I have prototyped is a kernel-core/kernel-drivers split. Here "core" could be analogous to "cloud", but I imagine it will be useful for other things. People wanting to do PCI passthrough can just install the kernel-drivers package.
Fair enough, agreed. We might need to eventually revisit this decision if SRIOV or graphic cards computing really take off and people complain. But right now, it's edge cases.
Agree, I don't think it makes much sense to split things into many small pieces.
- Do you need/want a firewall (requires iptables, etc)?
While not strictly necessary (the IaaS usually puts a firewall right in front of the instance / guest), I think we do want to allow each and everyone to optimize their security needs. The local iptables also has way more capabilities than the provided one offers.
I'd say yes by default, but being able to remove it might be useful (kernel-netfilter subpackage)?
So you agree multi-tiered subpackages is a bad idea, but then you propose a netfilter specific subpackage? ... Probably not. They'll likely just be in kernel-core.
- Do you need/want NFS or other cloudy storage things (for gluster?)?
Usually, you just attach additional volumes when you need more space but, particularly in private clouds, people do all kinds of weird things to get their data from here to there so I think we want all storage options. Also, support for every (major) filesystem that you could put on additionally attached disk drives.
I'm personally using NFS for host/guest filesystem sharing, so I'd vote yes.
- Do you need/want openvswitch?
I think no. That's a host side thing and this is about the guest kernel, right?
Nested virt? I dunno, it was mentioned at one point so I thought I'd bring it up for discussion again.
There are use cases that use OVS but I'd say they are rare. And nested virt is even rarer (particularly since Docker helped pushing lxc). So I don't think we want OVS in the guest image.
The list of modules I have in my local rawhide KVM guest is below. The snd_* related drivers probably aren't necessary. The btrfs and things it depends on can be ignored (unless you plan on switching from ext4).
As mentioned above, btrfs could be used on user-attached drives so keep that in there. But the root disk will be in line with the other Fedora products.
Depends on what is targeted. Strictly cloud? Or also someone running Fedora as a guest in virt-manager / boxes?
I'm of the opinion the latter is probably what we should shoot for. It's going to be the broadest target that is still reasonably small.
"Guest in virt-manager" is not part of our PRD, so it's not actually required. But as Matt mentioned, it's incredibly valuable for testing purposes. That should be a good enough reason to include something reasonably small.
I'd tend to include drivers for pretty much everything qemu can emulate.
<snip-snap detailed module talk>
OK. Was planning on that already.
Make it so :)
-- Sandro
kernel@lists.fedoraproject.org
-
Bruno Wolff III -
Don Zickus -
drago01 -
Gerd Hoffmann -
Josh Boyer -
Justin Forbes -
Kevin Fenzi -
Matthew Miller -
Reindl Harald -
Sandro "red" Mathys