On Thu, 24 Jun 2004 01:23:38 +0200, Axel Thimm axel.thimm@atrpms.net wrote:

On Wed, Jun 23, 2004 at 07:20:32AM -0400, Build System wrote:

...

kernel-2.6.7-1.448

• Mon Jun 21 2004 Arjan van de Ven arjanv@redhat.com

• make kernel-doc and kernel-sourcecode builds independent of eachother
• disable kernel-sourcecode builds entirely, we'll be replacing it with documentation on how to use the src.rpm instead for building your own kernel.

We just recently had the discussion of how many documented processes will get broken with the kernel-source -> kernel-sourcecode change. Not having it at all doesn't really improve that. ;)

What's the idea behind this? And what about people building external kernel modules? For the reasons outlined in the previous thread on this /lib/modules/`uname -r` is not adequate for that. Having the kernel-source package change its name is one thing we can adapt to (even though it was not necessary). Having it disappear for the face of earth is another :(

Please, get it back!

Axel.Thimm at ATrpms.net

While I'm far from an expert, the headers needed to build 3rd party modules are now packaged with the kernel proper (that's one reason it's taking so long to install a new kernel nowadays). The kernel-sourcecode package is sort of redundant as the actual source is in the kernel*.src.rpm anyway, but can be built (as i did) by editing the kernel-2.6.spec and changing the line near the top to say "build sourcecode=1" instead of 0.

On Wed, 2004-06-23 at 16:01 -0700, alan wrote:

It is going to make life more difficult for those who need to modify their kernel for whatever reason.

The same argument could be made for every other package we ship.

This does not appear to be a useful change. It makes it more difficult to get to the source code, not less.

Why was this change made? It seems counter-productive to me.

It makes the kernel just like _every other package in the distribution_. There's no reason why building a custom kernel should be considered to be any different from building, eg, a custom glibc. The "it's always been that way" argument doesn't really fly. Especially since once you understand making the changes from within the package, that's actually easier, more reproducible and makes your life easier for maintaining the changes you want to make over the long term (or even in working towards getting those changes included in the upstream kernel source.

Jeremy

...

There's no reason why building a custom kernel should be considered to be any different from building, eg, a custom glibc.

I'd bet good money on needing access to kernel source to build something (typically a driver, e.g. for the NVidia and centrino drivers)

you can't use kernel-sourcecode package for that, nor do you need it, so this part of your argument isn't correct.

or switch on-off some options being a couple of orders of magnitude more common than building a custom glibc.

which is one shell command away via rpmbuild -bp .... you need to enter a few dozen shell commands to compile a kernel anyway, is one more such a big deal ?

On Thu, 2004-06-24 at 07:46, Philip Balister wrote:

On Thu, 2004-06-24 at 04:06, Trond Eivind Glomsrød wrote:

...

I'd bet good money on needing access to kernel source to build something (typically a driver, e.g. for the NVidia and centrino drivers) or switch on-off some options being a couple of orders of magnitude more common than building a custom glibc.

Dunno about nvidia, but you do not need to install the kernel source to build he centrino drivers. They build fine with /lib/modules/...../build. The centrino drivers are almost trivial to build after a kernel upgrade.

Philip

Building the nvidia driver with the stock 2.6 kernel is a waste of time anyway. It will not run.

On Wed, Jun 23, 2004 at 07:44:15PM -0400, Chris Kloiber wrote:

On Thu, 24 Jun 2004 01:23:38 +0200, Axel Thimm axel.thimm@atrpms.net wrote:

...

On Wed, Jun 23, 2004 at 07:20:32AM -0400, Build System wrote:

...

kernel-2.6.7-1.448

• Mon Jun 21 2004 Arjan van de Ven arjanv@redhat.com

• make kernel-doc and kernel-sourcecode builds independent of eachother
• disable kernel-sourcecode builds entirely, we'll be replacing it with documentation on how to use the src.rpm instead for building your own kernel.

We just recently had the discussion of how many documented processes will get broken with the kernel-source -> kernel-sourcecode change. Not having it at all doesn't really improve that. ;)

What's the idea behind this? And what about people building external kernel modules? For the reasons outlined in the previous thread on this /lib/modules/`uname -r` is not adequate for that. Having the kernel-source package change its name is one thing we can adapt to (even though it was not necessary). Having it disappear for the face of earth is another :(

Please, get it back!

While I'm far from an expert, the headers needed to build 3rd party modules are now packaged with the kernel proper (that's one reason it's taking so long to install a new kernel nowadays).

In theory, yes, but try to build a kernel module for i686 on an athlon box. The kernel sources will clash with the running kernel.

Also it is quite an overshooting to have to install a foreign kernel for a simple build of a kernel module. Installing/Uninstalling the kernel will take more time than the kernel module build itself. But that could be solved by packaging the headers in separate (from the kernel) packages.

The kernel-sourcecode package is sort of redundant as the actual source is in the kernel*.src.rpm anyway, but can be built (as i did) by editing the kernel-2.6.spec and changing the line near the top to say "build sourcecode=1" instead of 0.

Of course, but if the kernel-source(code) gets considered obsolete and deprecated this will not stay for long there.

On Wed, Jun 23, 2004 at 09:43:54PM -0400, Jeremy Katz wrote:

On Thu, 2004-06-24 at 02:14 +0200, Axel Thimm wrote:

...

In theory, yes, but try to build a kernel module for i686 on an athlon box. The kernel sources will clash with the running kernel.

I'd rather fix this problem than paper over it with the kernel source. Especially since to really do things with the kernel source right now, you have to make changes to the tree in /usr/src.

Really, this is the main issue at stake. Us packagers have been building packages of kernel modules for years on 2.4. There is a defined way to use the installed kernel-headers to create modules that work with 1586, 1686, athlon, and the SMP variations thereon.

Now we have 2.6 and the same thing doesn't work and there's lots of confusion about how to build external kernel modules. I know I'm confused.

First, everything you need to build kernel modules is installed with the kernel package proper in /lib/modules/`uname -r`. No kernel source package is required. It just works, try it.

FC2 has an i586 and i686 kernel packages. The 2.6 kernel is smart enough to do the athlon optimizations on the fly, so that's one less arch to build for. SMP modules appear to be identical to non-SMP. (From what I've gathered...not sure if there should be a difference there or not.) How do you build for the i586? I'm not going to remove my running kernel to install the i586 arch.

The kbuild system supports cross compiling but I don't see anything to handle the sub-arches like i586 and i686.

Jack

On Thu, Jun 24, 2004 at 10:59:31AM -0400, Jeremy Katz wrote:

On Thu, 2004-06-24 at 10:29 -0400, Jack Neely wrote:

...

On Wed, Jun 23, 2004 at 09:43:54PM -0400, Jeremy Katz wrote:

...

On Thu, 2004-06-24 at 02:14 +0200, Axel Thimm wrote:

...

In theory, yes, but try to build a kernel module for i686 on an athlon box. The kernel sources will clash with the running kernel.

I'd rather fix this problem than paper over it with the kernel source. Especially since to really do things with the kernel source right now, you have to make changes to the tree in /usr/src.

Really, this is the main issue at stake. Us packagers have been building packages of kernel modules for years on 2.4. There is a defined way to use the installed kernel-headers to create modules that work with 1586, 1686, athlon, and the SMP variations thereon.

Now we have 2.6 and the same thing doesn't work and there's lots of confusion about how to build external kernel modules. I know I'm confused.

Bingo. The current situation, though, is far better for a single user who wants to build a kernel module. It loses some for packagers. Coming up with a solution that makes things easy for the user who just wants to build a module for their running kernel _and_ the person trying to package modules for a larger audience will be a big win.

...

First, everything you need to build kernel modules is installed with the kernel package proper in /lib/modules/`uname -r`. No kernel source package is required. It just works, try it.

Yep. Hooray, mkkerneldoth can die :)

...

FC2 has an i586 and i686 kernel packages. The 2.6 kernel is smart enough to do the athlon optimizations on the fly, so that's one less arch to build for. SMP modules appear to be identical to non-SMP. (From what I've gathered...not sure if there should be a difference there or not.) How do you build for the i586? I'm not going to remove my running kernel to install the i586 arch.

SMP modules are different from the non-SMP ones, but that's not a problem as you can install kernel-smp in parallel with your main kernel package. The i586 vs i686 part is the hard one.

Personally, I think that having a kernel-devel package that puts the headers somewhere with both the `uname -r` and arch as part of the path would be useful. That ends up working very nicely for people doing packaging. Unfortunately, it goes back to making things more difficult for an end-user. The best thing I've come up with at present for that is to continue to do like is being done now (headers for the current kernel in /lib/modules/$(uname -r)/build) and have kernel-devel be in addition to that. It would be a little bit higher disk space usage, but as kernel-headers would only need to be installed for special cases, I'm not sure that's a huge concern. Actually, I guess that /lib/modules/ $(uname -r)/build could be a symlink to /usr/include/kernel-headers/ $(uname -r)/$arch and be included as a broken symlink in the package. That _might_ work for keeping both camps happy.

I was thinking something on the lines of a kernel-devel package anyway. That would also solve the it takes to long to install the kernel package. This stuff is only needed when building things for the kernel and every other package would use a -devel subpackage.

Thoughts?

Jack

That's just my thinking out loud, though... I'm sure other people have ideas as well.

Jeremy

-- fedora-devel-list mailing list fedora-devel-list@redhat.com http://www.redhat.com/mailman/listinfo/fedora-devel-list

On Sun, 2004-07-04 at 13:52, Thomas Vander Stichele wrote:

...

...

Personally, I think that having a kernel-devel package that puts the headers somewhere with both the `uname -r` and arch as part of the path would be useful. That ends up working very nicely for people doing packaging.

Yep, this is what I've done and proposed as a general solution for fedora.us, and Freshrpms is using this as well now.

Where? I don't see any related devel or module packages in http://ayo.freshrpms.net/fedora/linux/2/i386/RPMS.freshrpms/

On Jun 24, 2004, Jeremy Katz katzj@redhat.com wrote:

SMP modules are different from the non-SMP ones, but that's not a problem as you can install kernel-smp in parallel with your main kernel package. The i586 vs i686 part is the hard one.

Not hard at all. Just add i586 to the kernel version number somewhere, and it will be a different pathname just like smp.

On Jun 24, 2004, Jeremy Katz katzj@redhat.com wrote:

On Thu, 2004-06-24 at 13:03 -0300, Alexandre Oliva wrote:

...

On Jun 24, 2004, Jeremy Katz katzj@redhat.com wrote:

...

SMP modules are different from the non-SMP ones, but that's not a problem as you can install kernel-smp in parallel with your main kernel package. The i586 vs i686 part is the hard one.

Not hard at all. Just add i586 to the kernel version number somewhere, and it will be a different pathname just like smp.

Yes, but that's ugly to look at :-)

So what? smp is ugly too, but it's there because it's useful. It fixes a real problem, and it doesn't hurt much, so...

On Thu, 2004-06-24 at 13:18 -0400, David T Hollis wrote:

On Thu, 2004-06-24 at 13:26 -0300, Alexandre Oliva wrote:

...

...

Yes, but that's ugly to look at :-)

So what? smp is ugly too, but it's there because it's useful. It fixes a real problem, and it doesn't hurt much, so...

This seems like a pretty workable approach that can make things work for all sides. I suppose Anaconda would need some tweaking so it knows to install kernel-i586 instead of plain ol' kernel but it already has that logic for handling the smp case.

Alexandre was suggesting in the version or release, not the name or even just in the uname output and not as part of the name. I still think that encoding the arch somehow that's visible to uname is kind of ugly, especially when you consider some of the arches (ppc64iseries anyone :)

As far as the difficulty of kernel-i586, having kernels named something other than 'kernel' makes things in anaconda _extremely_ difficult (and also then makes it hard to figure out what kernel someone has installed).

I'd love to see kernel-smp get folded into the main kernel package to be honest and as we move towards things like HyperThreading being present, it makes more and more sense. For the little bit of performance hit due to spinlock overhead, it would be good to get to where that can be minimized as much as possible and then everybody ends up being better off.

Jeremy

On Thu, Jun 24, 2004 at 03:40:54PM -0300, Alexandre Oliva wrote:

But it actually solves another problem. How do you tell otherwise which kernel arch you've booted into?

uname -m

Emmanuel

Sorry for my English.

Le jeu 24/06/2004 à 20:40, Alexandre Oliva a écrit :

On Jun 24, 2004, Jeremy Katz katzj@redhat.com wrote:

...

Alexandre was suggesting in the version or release, not the name or even just in the uname output and not as part of the name.

Right. Not part of the kernel package name, just part of the version string.

I am not sure if i understand all the problem. Here, i have three kernel base on the same kernel source : 2.6.6-1.435 : RH 2.6.6-1.435mat : RH without REGPARAM (bewan driver (binary) don't like this). 2.6.6-1.435matcustom : Custom configuration.

Simple, for each kernel configuration i have a different EXTRAVERSION (/usr/src/linux/Makefile).

The problem is that kernel.i586 and i686 use the same EXTRAVERSION and share file names.

Just add a "magic string" (the arch for example) in EXTRAVERSION for each configuration. That give : /boot/config-2.6.6-1.435-i586 /boot/System.map-2.6.6-1.435-i586 /boot/vmlinuz-2.6.6-1.435-i586 /boot/initrd-2.6.6-1.435-i586 /lib/modules/2.6.6-1.435-i586/*

No conflict. With this rule, at least kernel developers (freshrpms, etc) can add a kernel with "rpm -i --nodeps".

/lib/modules/`uname -r`/build should be in a -devel package. This save disk space.

If Fedora continue to ship kernel-source, add EXTRAVERSION in the directory name. Example : rename /usr/src/linux-2.6.6-1.435 to /usr/src/linux-2.6.6-1.435custom . /lib/modules/2.6.6-1.435custom/build should point to /usr/src/linux-2.6.6-1.435custom and not to /usr/src/linux-2.6.6-1.435 .

This is less confusing.

...

I still think that encoding the arch somehow that's visible to uname is kind of ugly

But it actually solves another problem. How do you tell otherwise which kernel arch you've booted into?

-- Alexandre Oliva http://www.ic.unicamp.br/~oliva/ Red Hat Compiler Engineer aoliva@{redhat.com, gcc.gnu.org} Free Software Evangelist oliva@{lsd.ic.unicamp.br, gnu.org}

Le ven 25/06/2004 à 14:25, Leonard den Ottolander a écrit :

Hello Matias,

...

/boot/config-2.6.6-1.435-i586

...

No conflict.

I would say the extra dash would break a lot of scripts. I am used to break of version/release at the second dash at the end. Not very nice to introduce inconsistency in this regard.

Also, wouldn't kernel-2.6.6-1.435-i686 be considered newer than kernel-2.6.6-1.435-i586?

I am not clear. I am not talking about version-release of rpm. Current rpm name/version/release can stay in place : Paquet : kernel-2.6.6-1.435.i586.rpm : uname -r : 2.6.6-1.435-i586 Paquet : kernel-2.6.6-1.435.i685.rpm : uname -r : 2.6.6-1.435-i686

Now you can install kernel-2.6.6-1.435.i586.rpm and kernel-2.6.6-1.435.i685.rpm (perhaps with "rpm -i --nodeps" but it's not a problem for kernel developers).

Leonard.

Am Do, den 24.06.2004 schrieb Jeremy Katz um 16:59:

On Thu, 2004-06-24 at 10:29 -0400, Jack Neely wrote:

...

On Wed, Jun 23, 2004 at 09:43:54PM -0400, Jeremy Katz wrote:

...

On Thu, 2004-06-24 at 02:14 +0200, Axel Thimm wrote:

...

In theory, yes, but try to build a kernel module for i686 on an athlon box. The kernel sources will clash with the running kernel.

I'd rather fix this problem than paper over it with the kernel source. Especially since to really do things with the kernel source right now, you have to make changes to the tree in /usr/src.

Really, this is the main issue at stake. Us packagers have been building packages of kernel modules for years on 2.4. There is a defined way to use the installed kernel-headers to create modules that work with 1586, 1686, athlon, and the SMP variations thereon.

Now we have 2.6 and the same thing doesn't work and there's lots of confusion about how to build external kernel modules. I know I'm confused.

Bingo. The current situation, though, is far better for a single user who wants to build a kernel module.

I thought it worked fine in most situations before. But I also think the basic approach is better now.

It loses some for packagers.

Yes. Really.

Coming up with a solution that makes things easy for the user who just wants to build a module for their running kernel _and_ the person trying to package modules for a larger audience will be a big win.

++

...

FC2 has an i586 and i686 kernel packages. The 2.6 kernel is smart enough to do the athlon optimizations on the fly, so that's one less arch to build for. SMP modules appear to be identical to non-SMP. (From what I've gathered...not sure if there should be a difference there or not.) How do you build for the i586? I'm not going to remove my running kernel to install the i586 arch.

SMP modules are different from the non-SMP ones, but that's not a problem as you can install kernel-smp in parallel with your main kernel package. The i586 vs i686 part is the hard one.

Why can't they also contain the arch-part (i586, i686) like they contain the smp-tag. So they were installable all in parallel. With Suse that is possible. I like that very much.

And please don't answer "Yes, but that's ugly to look at :-)" ;-)

Yes, I know, for most people this in unimportant. And changing machines i686 and i586 happens not that often. That happened much more often when there was a special athlon kernel...

Personally, I think that having a kernel-devel package that puts the headers somewhere with both the `uname -r` and arch as part of the path would be useful. That ends up working very nicely for people doing packaging. Unfortunately, it goes back to making things more difficult for an end-user. The best thing I've come up with at present for that is to continue to do like is being done now (headers for the current kernel in /lib/modules/$(uname -r)/build) and have kernel-devel be in addition to that. It would be a little bit higher disk space usage, but as kernel-headers would only need to be installed for special cases, I'm not sure that's a huge concern. Actually, I guess that /lib/modules/ $(uname -r)/build could be a symlink to /usr/include/kernel-headers/ $(uname -r)/$arch and be included as a broken symlink in the package. That _might_ work for keeping both camps happy.

I would really like a solution like that. In fedora.us there ist a package in the QA queue with contains the files from /lib/modules/$(uname -r)/build of all different kernels in /usr/src/. See

http://bugzilla.fedora.us/show_bug.cgi?id=1709

That's just my thinking out loud, though... I'm sure other people have ideas as well.

BTW: IMHO we should also consider if we use the approach (at hole or in parts) that the kernel-developers are working on. See

http://www.uwsg.indiana.edu/hypermail/linux/kernel/0406.2/1280.html

Arjan also commented in the thread but was not that enthusiastic AFAICS... But I don't what to end in a situation where building external modules is different then the kernel documentation suggest. Or different from all other distributions...

Just my 2¢

CU thl

On Fri, 2004-06-25 at 14:18, Leonard den Ottolander wrote:

This rather fundamental change has not even been pre announced. This has nothing to do with the way a community project should work.

which fundamental change? the fact that you can't use kernel-source(code) to build external modules? That has been the case for all the 2.6 rpms, and is a result from the 2.6 buildsystem changes more than anything else, and was there even before the very first fc2 test release.

On Fri, Jun 25, 2004 at 03:04:24PM +0200, Leonard den Ottolander wrote:

Hi Arjan,

...

which fundamental change? the fact that you can't use kernel-source(code) to build external modules? That has been the case for all the 2.6 rpms, and is a result from the 2.6 buildsystem changes more than anything else, and was there even before the very first fc2 test release.

I would call the fact that you drop the kernel-source(code) rpm

^^ propose to ^^

altogether, from which people have been building custom kernels for years a "fundamental change".

yes and that's why it need to be testable, tested, discussed etc etc and that is why it's not done irreversibly. I am still behind the proposal to drop the package assuming we can document custom kernel building well enough. (Both in the release notes and in kernel-doc I guess, maybe even a text file in /usr/src/linux-<foo>/ or a script there). We're at 4 binary cd's already, if we're not careful about bloat we'll be at 12 in no time.

Also, the compiling for different architecture issue should have been addressed before making the change.

Several solutions have been floated on this list over the last few months. The one I personally like most is the concept that the openafs rpm, that was posted here the other week, uses. That solution has the quite big advantage of allowing users of this approach to make your modules for *all* released kernels in one go and in one package, and thus allowing the user to go back to older kernels without having to worry about an additional extra package needed for that. Another person also had a script to build modules automated, but done in a different way, so I would not call this an issue without solution at all.

On Fri, 2004-06-25 at 16:32, Jack Neely wrote:

...

Several solutions have been floated on this list over the last few months. The one I personally like most is the concept that the openafs rpm, that was posted here the other week, uses. That solution has the quite big advantage of allowing users of this approach to make your modules for *all* released kernels in one go and in one package, and thus allowing the user to go back to older kernels without having to worry about an additional extra package needed for that. Another person also had a script to build modules automated, but done in a different way, so I would not call this an issue without solution at all.

No. Including kernel headers in every single package that builds an external module is *WRONG*. Not to mention the side effect of building, say, openafs modules for, how many kernel errata does FC1 have now?

surely someone can package it fully separate? I would actually prefer to build for all errata. There's only a dozen max or so per release...

I've talked with enough people and read enough emails and documentation to understand that there is a pretty excepted standard for packaging kernel modules.

All the ones I've seen have problems with the "want to have multiple kernels and be able to go back and forth without pain" case. Oracle's OCFS does this similar (and auto-picks the right architecture out of it's archives of built modules) and that Just Works(tm). Users can go back and forth all they want, install older kernels, install other architectures of older kernels.. it Just Works(tm).

kernel-smp is merged into kernel

well.... the problem here is that a significant number of x86 class machines don't boot the smp kernel....

Jeff Spaleta wrote:

. I think having a kernel-devel subpackage makes a lot of sense, especially if the goal is to treat the kernel package similarly to other packages. I dont need to compile modules on most of the computers I admin, so I dont need the associated bits.

Sorry also for coming late to this thread, but I'd like to endorse kernel-devel as well. It could provide the /lib/modules/$(uname -r)/build stuff as well as the /usr/src/linux stuff which is currently in kernel-source. Also it could make sure that /lib/modules/$(uname -r)/build is a symlink to the appropriate place in /usr/src/linux which would make building legacy modules easier.

So, my proposal is:

Fold kernel-source into kernel-devel; make kernel-devel have Provides: kernel-source to make BuildRequires in older kernel src rpms easier to live with, and keep it as part of Fedora Core.

Alexandre Oliva wrote:

On Jun 25, 2004, Aaron Bennett aaron.bennett@olin.edu wrote:

...

Sorry also for coming late to this thread, but I'd like to endorse kernel-devel as well.

I'd like that too. There's no reason to keep headers around if I'm not going to build modules, and don't *want* to accidentally build modules.

It shouldn't contain full kernel sources, though (they're available elsewhere), only whatever is needed to build kernel modules.

This will not only save space, but also install time, since hardlinks will likely run faster if it only looks at the modules tree.

The only (minor) issue with splitting kernel-devel out of kernel is that up2date should probably be taught to not upgrade this package by default, otherwise you may end up being unable to build modules for the running kernel just because you up2dated your box to a newer kernel release, which brought in a new kernel-devel.

You would just need to reconfigure up2date to do an install instead of update, ie.:

pkgsToInstallNotUpdate=kernel;kernel-devel;

-- Brian Gerst

I'll throw this out as a straw man based on what I've been reading:

1) /usr/lib/`uname -r`/build continues to exist as part of the kernel package. This covers what most people who do want to build modules for their currently running kernel.

2) A new kernel devel package is created. It includes headers for ALL kernels that have been released for a distribution. They'd be placed in different directories (maybe /usr/src) with some naming scheme that includes kernel release, smp, architecture, etc. This package would get new sets of headers added to it every time a new kernel release comes out.

Pros: ----- * For most users #1 is sufficient, small, and easily automated. * For people building drivers for distribution #2 covers everything they might need without requiring them to keep reconfiguring a kernel tree. * #2 is probably smaller than kernel-source even with a bunch of kernel versions, but this should be checked. The distribution CD is definitely small because it only has one set of headers. * #2 can be a normal package with version numbers that is probably the same as the kernel version numbers. * #2 can be safely installed without screwing up a running system because #1 always exists and #2 installs in a different tree. * There's only a little duplication (one header tree) between #1 and #2. * #2 has no issues with SMP kernels on non SMP machines. * Even if third parties make new kernel packages they can also release kernel-devel that is a includes everything from the stock kernel-devel and adds their header tree under a unique name. * Kernel modules probably don't need a requirement on this package because they'll be building based on /lib/modules/`uname -r`, but they could if they want everything. * Kernel module builders can loop over all the header trees and build every possible configuration.

Cons: ----- * Creating #2 is a bit tricky. The headers are essentially generated by kernel package builds. I'd think you'd add new source tarballs to the source RPM that contain the headers for the next batch of kernels. It's an additional step after building all the kernel packages, but this grief is limited to the building of kernel packages so it's only done once and everyone benefits afterward.

* The kernel-devel package keeps getting bigger. Someone should do some calculations based on FC1 to figure out how bad this will be.

How does this do meeting everyone's requirements? Shoot it full of holes. I'm just an neutral 3rd party who's been reading this thread for way too long. :)

- |Daryll

Correct, kernel-devel should only contain headers/kbuild for its matching kernel. Multiple kernel-devel packages should be able to be installed at the same time.

Jack Neely

On Fri, 2004-06-25 at 15:08, Daryll Strauss wrote:

2. A new kernel devel package is created. It includes headers for ALL

kernels that have been released for a distribution. They'd be placed in different directories (maybe /usr/src) with some naming scheme that includes kernel release, smp, architecture, etc. This package would get new sets of headers added to it every time a new kernel release comes out.

In a kernel-source*.rpm disabled world, this would mean adding headers unneeded for compiling the _kernel_ to the SRPM. Since installing the SRPM seems to be Arjan's proposed method of compiling a custom kernel w/ Red Hat patches I think that an SRPM that is bloated in this manner should be avoided.

-Toshio

On Sun, 2004-06-27 at 08:49, Matias Feliciano wrote:

Le dim 27/06/2004 à 13:51, Toshio a écrit :

...

On Fri, 2004-06-25 at 15:08, Daryll Strauss wrote:

...

2. A new kernel devel package is created. It includes headers for ALL

kernels that have been released for a distribution. They'd be placed in different directories (maybe /usr/src) with some naming scheme that includes kernel release, smp, architecture, etc. This package would get new sets of headers added to it every time a new kernel release comes out.

In a kernel-source*.rpm disabled world, this would mean adding headers unneeded for compiling the _kernel_ to the SRPM.

Are you sure ?

I could have misunderstood Daryll, but it seems that he's saying kernel-devel would have kernel headers generated for all arches for each kernel in a particular release. So for every errata kernel the headers from each of the previous errata (back to the first kernel in the release) would have to be tarred up (by the packager) and included in the new kernel SRPM. If you understood it differently, please let me know.

...

Since installing the SRPM

I use src.rpm but I don't install src.rpm. I never removed a src.rpm with "rpm -e".

You can't remove an SRPM with rpm -e because it's not entered into the rpm database. So in the sense that rpm tracks the contents of the SRPM you're right, it doesn't get installed. I don't always use rpm to install things on my system, however. OTOH, it's mostly programs I'd associate with "installing" and use "unpacking" or "untarring" with source.

I think this points to the fact that I don't think of the kernel source as "source". In my mind it's more of an equivalent to sendmail-cf or system-config-*; a package that aids in configuring another package. I'm sure I'll be able to adjust to using the SRPM to "configure" my kernel but the idea that removing kernel-source makes the kernel package more consistent with the rest of the packages in the OS is counterintuitive to me because of this conceptual difference.

-Toshio

Toshio wrote:

I see Arjan did understand it differently. Thanks for the clarification, it makes more sense that way (kernel-devel is built from a separate, kernel-devel SRPM.).

I'm dense.... why exactly would it make sense to have a seperate kernel-devel SRPM? Does any other -devel package for a piece of core come from an a seperate -devel SRPM?

Let's recap again.... the potential benefits so far stated for dropping the kernel-sourcecode binary have been: 1) Reduced bloat in Core, to prevent shipping and mirring the sourcecode when the srpm can be used to rebuild a custom kernel using the packaging methodology of srpms 2) Treating the kernel packages just like all the other packages in Core.

The ONLY known problem with removing kernel-sourcecode, is dealing with how to build extra modules for multiple arches for binary kernels by just installing binary packages, something public and intranet repositories need a clean solution for. But there's never really been a clean solution for this.

One potential solution to the problem of add-on module building repository style which is in line wht the potential benefits as stated, is reworking the kernel packaging to create a kernel-devel package. If we want to make the kernel package as similar to other packages as possible, kernel-devel should be used to hold the bits needed to compile add-on modules for the kernel, instead of the kernel package itself. But I see no reason for kernel-devel package to hold information for ALL the previous released kernels as well. I just don't see why thats needed. If kernel binaries across versions can be installed in parallel the associated kernel-devel from different kernel version should be parallel installable as well as a matter of design.

The only problem with using a kernel-devel package is the same problem the kernel binary suffers from right now, dealing with situations like parallel installs of i586 and i686. The only potential solution to this problem has been to have kernel-devel include build information for ALL arches for the given kernel release. Several people have brought this up, but I've seen very little to no discussion on the potential downsides to piling in all the arches into one kernel-devel.

-jef

On Sun, Jun 27, 2004 at 07:20:30PM +0200, Arjan van de Ven wrote:

Having a separate kernel-devel src.rpm which is made *after* the build takes all this mess away since it can just take the right headers from the actual packages.

After _which_ build? Will you be snatching the files from the "final" beehive build or some test build that differs from the final one?

In other words, is the kernel-devel package consistently reproducible? Mirek

On Sun, Jun 27, 2004 at 07:20:30PM +0200, Arjan van de Ven wrote:

...

The only problem with using a kernel-devel package is the same problem the kernel binary suffers from right now, dealing with situations like parallel installs of i586 and i686. The only potential solution to this problem has been to have kernel-devel include build information for ALL arches for the given kernel release. Several people have brought this up, but I've seen very little to no discussion on the potential downsides to piling in all the arches into one kernel-devel.

well you first would need to show it's possible and future proof....

right now we put the info in /lib/modules/../build *after* building that exact kernel, and there's a good reason for that: the kbuild system assumes a fully built tree. Now in the rpm we cheat and effectively remove the files we don't need. And *maybe* now that doesn't depend too much on the build results, but if you enable modversions for example it already does.

In what way? What files would be missing?

And some of the kbuild proposals on lkml also go in the general direction of not being able to get all the needed info without doing a full build.....

Having a separate kernel-devel src.rpm which is made *after* the build takes all this mess away since it can just take the right headers from the actual packages.

Well, since you do have a full build at kernel package creation time the current procedure seems fine, only the final packaging needs to be changed, e.g. introduce kernel-headers/devel[-smp|-<other flavour>] packages on par with the kernel[-smp|-<other flavour>] packages, and have the latter have a symlinked /build/ to the former.

Each kernel has its own kernel-devel/header, and yum/apt/up2date etc need to be tought to handle these new multiple packages just like "kernel" itself.

You don't want to shove all headers in one rpm, as this will again make it harder to build custon kernels with their custom kernel-headers/devel packages.

On Mon, Jun 28, 2004 at 10:10:41AM +0200, Arjan van de Ven wrote:

On Mon, Jun 28, 2004 at 09:15:49AM +0200, Axel Thimm wrote:

...

On Sun, Jun 27, 2004 at 07:20:30PM +0200, Arjan van de Ven wrote:

...

right now we put the info in /lib/modules/../build *after* building that exact kernel, and there's a good reason for that: the kbuild system assumes a fully built tree. Now in the rpm we cheat and effectively remove the files we don't need. And *maybe* now that doesn't depend too much on the build results, but if you enable modversions for example it already does.

In what way? What files would be missing?

the generated .mod.c and .mod.o files (for example jbd.mod.c and jbd.mod.o for jbd.ko module) would be missing for example.

OK, the kernel %install script could detect whether modversions have been selected in .config and package these files, too, then. Will be quite a bloat, but if there is no other way...

...

packages on par with the kernel[-smp|-<other flavour>] packages, and have the latter have a symlinked /build/ to the former.

I do not want /build/ to become a symlink again.

How will you solve the issues with a) same uname -r for different kernels (different archs)? b) splitting kernel headers for the kernel?

For a) you could start adding archs to EXTRAVERSION, for b) you could only skip the build folder/symlink altogether, or have kernel-headers/devel install into there. Given the possible bloat these would uneccessarily occupy needed space on /.

...

You don't want to shove all headers in one rpm, as this will again make it harder to build custon kernels with their custom kernel-headers/devel packages.

no it's not harder, they just provide their add-on kernel-devel package, so your kernel series could have a kernel-axel-devel rpm ...

No, it is ;)

The idea is to have kernel module src.rpms with

Requires: kernel-devel >= 2.6.0

and have the external build-system provide the matching rpms and --define 'kernelsrcdir /a/b/c' for which path to chose for building kernel modules against.

Having kernel module specfile for each kernel series defeats the purpose of specfile invariance across kernels.

You also want to provide users with a uniform way to build their own kernels and kernel-headers/devel packages, so you don't have much choice than to do it per kernel and not bundled.

On Mon, Jun 28, 2004 at 10:32:53AM +0200, Arjan van de Ven wrote:

On Mon, 2004-06-28 at 10:22, Axel Thimm wrote:

...

...

...

In what way? What files would be missing?

the generated .mod.c and .mod.o files (for example jbd.mod.c and jbd.mod.o for jbd.ko module) would be missing for example.

OK, the kernel %install script could detect whether modversions have been selected in .config and package these files, too, then. Will be quite a bloat, but if there is no other way...

exactly, but it can only do that AFTER building the kernel. So the plan of one grand unified kernel-devel for all architectures isn't a feasible one.

So have a list of required files be created at the end of %build. And a GUT of kernel-devel is a bad idea, really!

...

...

...

packages on par with the kernel[-smp|-<other flavour>] packages, and have the latter have a symlinked /build/ to the former.

I do not want /build/ to become a symlink again.

How will you solve the issues with a) same uname -r for different kernels (different archs)?

that's unrelated entirely.

So where will you put kernel headers for different arches, if you don't redesign this part? You either have to change uname -r or blow away the concept of non-symlinked /build/. I don't think that is unrelated, at all.

...

b) splitting kernel headers for the kernel?

I don't see splitting as a goal. Providing separate we can talk about, but splitting I don't see as option; the current mechanism has too many advantages for that.

What advantages? That you need to install a kernel only for building kernel modules for it? And that this kernel may even conflict with the running one?

Split the headers off the kernel rpms, you will be doing yourself, users and packagers a great favour. There are no advantages in either bundling all header files together, nor in bundling kernel and kernel header files.

...

...

...

You don't want to shove all headers in one rpm, as this will again make it harder to build custon kernels with their custom kernel-headers/devel packages.

no it's not harder, they just provide their add-on kernel-devel package, so your kernel series could have a kernel-axel-devel rpm ...

No, it is ;)

The idea is to have kernel module src.rpms with

Requires: kernel-devel >= 2.6.0

and your kernel-axel-devel can't Provides: kernel-devel = <foo> ?

Recent (FC1 upwards) versions of rpm define versioned Provides as to be uniqified, so no, apt/yum/rpm and so on will chose to only have one of them installed (It has been filed as a bug and closed with consdiered as defined behaviour).

It is also the intention to have a _uniform_ way of packaging kernels and kernel modules, and not having 3rd parties and users do kludgy stuff again, right?

...

You also want to provide users with a uniform way to build their own kernels and kernel-headers/devel packages, so you don't have much choice than to do it per kernel and not bundled.

I disagree with that conclusion. It's one simple script as the openafs rpms and the other method posted here both have shown.

And you have read the comments on that.

Please do trust the people that have been doing the packaging of several kernel module rpms for some time now. We have learned to cope with the shortcomings of the current kernel-source(code) methods, and since we now have the opportunity to do it right, we should do so.

Just check a few kernel module rpms yourself to see the real problems in the dirt and not on the blueprints ;)

The demand profile is o kernel module rpm should be agnostic of location of kernel headers/source. This is passed with "--define"s to the src.rpm. o They should be independent of RH, ATrpms, other 3rd party repos, custom kernel rpms, e.g. one specfile/src.rpm for all. The choice is driven by the --define above o Allow for arbitrary flavours (up, smp, "custom", old flavours like bigmem, new flavours). flavour is also passed from outside. o header files/source may not overlap for different arch/flavour combination.

On Mon, Jun 28, 2004 at 12:40:22PM +0200, Arjan van de Ven wrote:

actually it 100% is. kernel-devel, if it ever comes to a reasonable thing, will be in *addition* to what is there right now not as replacement.

Maintaining two solutions for the same problem? It is per definition error-prone and predestined that one of them will deteriorate over time.

...

Split the headers off the kernel rpms, you will be doing yourself, users and packagers a great favour. There are no advantages in either bundling all header files together, nor in bundling kernel and kernel header files.

no splitting this off will hurt users. Maybe not packers but users it does.

How will this hurt users, can you explain?

I'm sorry but I have a really hard time taking packaging advice from anyone who uses the current kernel-sourcecode method to build module rpms.

I am sorry you feel that way, and I can only verify that the kernel-source(code) package is flawed in this respect. But it is rectifyable and better than not having an rpm to depend on at all.

And after all, we are trying to fix the current methods of kernel source/headers deployment, so hacking on your own package design won't help you. It was all you gave us, and we did our best in the given framework.

And practice shows that it has worked, kernel modules are downloaded, installed and successfully used at a very great extent. You cannot ignore that, no matter how hard you try. ;)

...

Just check a few kernel module rpms yourself to see the real problems in the dirt and not on the blueprints ;)

I've looked at some and I have yet to find a single one that is packaged remotely correctly. (if that is at all possible, something I'm not entirely convinced of)

It is, perhaps I'll find time to fix the kernel src.rpm over the weekend to demonstrate. Too bad it has close to zero chances of being adopted, judging from this thread. ...

...

The demand profile is o kernel module rpm should be agnostic of location of kernel headers/source. This is passed with "--define"s to the src.rpm.

this basically makes the point moot of where the kernel headers come from. If you have to pass the location in the name of the exact requires: is utterly irrelevant surely.

No, no macros into BuildRequires, all you have is

BuildRequires: kernel-headers >= 2.6.0 (In a previous main I had written Requires instead of BuildRequires).

Of course you will use variable Requires in the resulting binary kernel module rpms to match the targeted kernel.

...

o They should be independent of RH, ATrpms, other 3rd party repos, custom kernel rpms, e.g. one specfile/src.rpm for all. The choice is driven by the --define above

... and the --define can't give the exact package to require for the headers? I don't believe that.

No, because that would be dependency information embedded into the src.rpm, which is to be kept independent of the kernel.

...

o header files/source may not overlap for different arch/flavour combination.

I think you mean "it must be possible to get all headers for all archs/flavors installed in parallel somehow". That is a quite less restrictive requirement.

Well, let's agree in that one to have something to agree with ...

Anyway I thank you for this list of requirements, as far as I can see there's nothing in there that voids or blocks the kernel-devel mechanism I proposed, in fact it strengthens it...

OK, I am getting wierd off in trying to keep you from jumping off the cliff. Just do it and see for yourself. I now understand, why you need social experiments in rawhide. ...

On Mon, Jun 28, 2004 at 10:42:53AM -0700, Per Bjornsson wrote:

Now, I could agree that given a kernel-devel package it would be possible to build any correctly written module without including the files again in /lib/modules/`uname -r`/build, but that is the officially documented way of doing it and it would be a shame to break the official convention. The alternative would be a symlink of course, but then you're forcing everyone who wants to do a quick module build to include the stuff needed for all architectures (since that's what's in the hypothetical kernel-devel package).

But that is exactly the point of discussion. A kernel-devel package should be per kernel version, arch and flavour, not a conglomerate. That way you both have less bloat and it is universal to be extended to arbitrary kernel, e.g. ones in future errata, custom kernels and so on.

The all-errata-in-one package would need extra maintainance from a central place, and would only allow building for the given set of kernels.

So with kernel-2.6.8-9.8.7.i686.rpm you get a symlink

/lib/modules/2.6.8-9.8.7/build -> /usr/src/kernel-headers/2.6.8-9.8.7.i686

And kernel-devel-2.6.8-9.8.7.i686.rpm contains the headers in /usr/src/kernel-headers/2.6.8-9.8.7.i686

A kernel module src.rpm BuildRequires kernel-devel >= 2.6.0, you drop one (or more) of the various kernel-devel packages and point the rpmbuild to it like

rpmbuild --rebuild --define 'kernel 2.6.8-9.8.7' --target i686 foo-kmdl-1.2.3-4.5.6.src.rpm (perhaps the target option could even be skipped)

and this builds

kernel-module-foo-2.6.8-9.8.7-1.2.3-4.5.6.i686.rpm

That's all, folks!

But that is exactly the point of discussion. A kernel-devel package should be per kernel version, arch and flavour, not a conglomerate. That way you both have less bloat and it is universal to be extended to arbitrary kernel, e.g. ones in future errata, custom kernels and so on.

You have said this over and over but not provided any explanation on why this is so. "should be" is your personal opinion.

less bloat: installing four kernel rpms worth of content to build for four combinations of kernel takes over 50 MB. I don't see how that's less bloat. A tradeoff symlinked forest like I have only adds 1.5 MB on top of one kernel installed, which compared to 50 MB is a very good tradeoff. On top of that, spec files can *still* be written *without any additional -devel package* for the single-kernel user case. I consider your solution more bloat.

"universal to be extended to arbitrary kernel": maybe you should give an example, because I really don't understand what you're trying to say here. I don't see how Red Hat would be suddenly unable to have a one-size-fits-all devel package for all kernels they release as a set, since they are releasing the set. Same goes for outside people. Seems like a strawman's argument to me.

The all-errata-in-one package would need extra maintainance from a central place, and would only allow building for the given set of kernels.

I don't see how creating *four* rpms for four kernels as compared to *one* devel rpm for four kernels is extra maintenance.

People releasing their custom kernel in the wild should take up responsibility and provide the same mechanism as upstream does.

Thomas

Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> and it looks so pretty all those tiny bright lights calling my name <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

Hi,

Thomas, I understand the thread is long and old by now, but this kind of statements are completly moot and only to be considered as heat-ups. I have given grounds to all the above in past and today's posts. I'd like to ask you to check the thread again, as this is the third (!) post from you having me presented as a hot-air mouthed troll.

Hm, I have not used such wording at all and your interpretation is very subjective. I refuse to be lumped in with the "we think axel is a troll"-crowd just because I state my opinion. Believe me, you'll know when I'm trolling :)

...

"universal to be extended to arbitrary kernel": maybe you should give an example, because I really don't understand what you're trying to say here.

OK, check my recent nvidia-graphics drivers that have been build out of the same src.rpm for the following 30 kernels:

2.4.20-35_39.rh7.3.at 20-35_39.rh7.3.atbigmem 20-35_39.rh7.3.atsmp 2.4.20-35_39.rh8.0.at 20-35_39.rh8.0.atbigmem 20-35_39.rh8.0.atsmp 2.4.20-35_39.rh9.at 20-35_39.rh9.atbigmem 20-35_39.rh9.atsmp 20-35.7 2.4.20-35.7bigmem 20-35.7smp 20-35.8 20-35.8bigmem 20-35.8smp 20-35.9 2.4.20-35.9bigmem 20-35.9smp 22-1.2197.nptl 22-1.2197.nptl_51.rhfc1.at 2.4.22-1.2197.nptl_51.rhfc1.atsmp 22-1.2197.nptlsmp 2.4.26-1.ll.rhfc1.ccrma 26-1.ll.rhfc1.ccrmasmp 2.6.6-1.427 2.4.2.6.6-1.427smp 2.6.6-1.435.2.3 2.6.6-1.435.2.3smp 2.4.2.6.7-1.456_4.rhfc2.at 2.6.7-1.456_4.rhfc2.atsmp

It includes vendor (RH) kernels, ATrpms kernels, and even PlanetCCRMA kernels. These are all the common kernels packaged and deployed BTW.

All this using a non-packaged version of the kernel-headers per kernel flavour/arch as descibed above. I'd call that "universal to be extended to arbitrary kernels", wouldn't you?

Yes, that's a good example. Also, I don't really see what that has to do with what Red Hat should be doing. They ship four kernels, and whether they ship four seperate -devel packages or one -devel for all four has no bearing on whatever you can do for all the kernels you ship, does it ?

And I'd also say the method has already proven in the field. There two dozen kernel module rpms successfuly deployed at ATrpms.net in the above manner resulting in several thousands (!!!) of binary kernel modules rpms.

So, yes, it's quite more than hot air ...

"hot air" is a word I never use.

...

I don't see how creating *four* rpms for four kernels as compared to *one* devel rpm for four kernels is extra maintenance.

Ehm, how will the user with his custom rpm be treated? Open the conglomerated rpm and add his own kernel? No, that's far from user-friendly. And the discussion of keeping things bundled is done in favour of the user, right?

A user with a custom rpm just builds whatever he wants against it. He doesn't even need a -devel package we're discussing about right now. I think you're drawing away the real discussion with these arguments.

And don't forget, the suggestion of one kernel header package per kernel/flavour/arch is producing those _automatically_ while building the kernels themselves. There is nothing you need to _do_ to get them, e.g. no extra maintenance, while adding a new kernel header to the mega-kernel-header package is extra maintainance.

Yes, that would be nice as a feature. It could still be done automagically whatever way is used however. You're talking about something different.

...

People releasing their custom kernel in the wild should take up responsibility and provide the same mechanism as upstream does.

You mean each _user_ who wants to have his custom kernel packaged and kernel modules build for it should become an rpm expert? Not really.

No, I mean that each person who provides custom kernels to others is expected to know about packaging. Otherwise he shouldn't be releasing ANY .rpm in the first place, let alone something as complex as a kernel one.

Thomas

Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> resistance is low when I'm feeling bored what I thought was fun isn't fun anymore <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

Hi Axel,

On Mon, 2004-06-28 at 12:58, Axel Thimm wrote:

On Mon, Jun 28, 2004 at 12:40:22PM +0200, Arjan van de Ven wrote:

...

actually it 100% is. kernel-devel, if it ever comes to a reasonable thing, will be in *addition* to what is there right now not as replacement.

Maintaining two solutions for the same problem? It is per definition error-prone and predestined that one of them will deteriorate over time.

I'd like this situation resolved just as much as you do, but I'm not always following what you're trying to say.

Please let's keep it constructive. Which "two solutions" are you saying Arjan is hinting at ? AFAICT he means something that is installed "on top of" the current kernel stuff.

You make a blanket statement shrouded in mystery then based on that preposition make a true, but irrelevant, broad sweeping statement. The point should be made on the truthfulness of the assumption, not the result :)

And after all, we are trying to fix the current methods of kernel source/headers deployment, so hacking on your own package design won't help you. It was all you gave us, and we did our best in the given framework.

I agree with this as well; Arjan has commented a lot on what we WERE NOT supposed to do in the past, but I'm pretty sure there is no viable solution Arjan can propose that we could have invented instead in the 2.4 period. I respect Arjan's work, but really, insulting us as a community about why we used the kernel-source rpm without ever having given us any suggestions on what we should have done instead is not very constructive either :)

Arjan, please, if there is something we should have done instead that satisfies the obvious requirements, then please say so; if not, then please don't ever ridicule outside people again for working around the broken packages we got from Red Hat.

In fact, I'm pretty sure if there had been some sort of open process where we worked together *before* the facts we could have gotten a nice solution for us all. Let's focus on that for the next iteration ?

Thomas

Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> Are you happy where you're sleeping ? Does he keep you safe and warm ? Does he tell you when you're sorry ? Does he tell you when you're wrong ? <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

Hi,

you are jumping quite late into this thread, so you may be missing the more detailed discussions on the different propositions made. There is no mystery or blanket statement.

Actually, I jumped quite soon into this thread. I have posted more than a few emails before on this list about my proposal for a general solution that works for 2.4 and 2.6 kernels, with various examples and requests for comments.

I'd like one solution as proposed on anothe post in the thread, that would place the bits required to develop kernel modules in unique folder names, say under /usr/src/.

/usr/src is the wrong place, but I assume you agree with that already.

Unique in the sense that the required bits for different archs should not overlap like they do now for i686 vs i586 vs x86_64 etc.

Yes, and everyone interested in packaging has already brought this up countless times, and Arjan has said countless times that he doesn't agree and it's not upstream kernel policy. We all know those arguments, and to everyone it seems quite clear that the only way we as packagers will be able to change this is to convince upstream kernel people.

Thomas

Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> There's nothing I want to see There's nowhere I want to go <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

Hi, again,

On Sun, Jul 04, 2004 at 01:13:59PM +0200, Thomas Vander Stichele wrote:

...

The idea is to have kernel module src.rpms with

Requires: kernel-devel >= 2.6.0

and have the external build-system provide the matching rpms and kernel modules against.

The build system does not need to provide kernelsrcdir if the location where the build files are stored has a decent name.

It does, because you may have installed several kernel headers for different kernels, so you'd like to pass to the build process for which kernels to build kernel modules. New kernels get added and old ones removed, and the src.rpm & specfile is to be kept invariant over time (unless other changes require patching etc.)

The spec file already has all the info to distinguish what type and arch to build for (up/smp and i586/i686), so just give sensible names to the path and it's solved.

How does the specfile know, whether I want to build up/smp and i586/i686 etc on my x86_64 smp build host that produces all kernel modules archs and flavours for all supported distros?

You could use defaults, for the running kernel, or the installed kernel(s), of course, but in general you would need that information to be an external input to the build process.

...

You also want to provide users with a uniform way to build their own kernels and kernel-headers/devel packages, so you don't have much choice than to do it per kernel and not bundled.

Nope, already works, and they're bundled.

Not sure I follow, my remark was wrt to a suggested mega-kernel-devel package containing kernel development bits for the N Red Hat kernel errata in one package, e.g. bundling the kernel headers for kernels

2.6.5-1.358 2.6.6-1.427 2.6.6-1.434 2.6.6-1.435 2.6.6-1.435.2.1 2.6.6-1.435.2.3

Each new kernel errata would require updating of this package, which may be fine for RH, but not for custom kernels that will be missing in the above package (e.g. a 2.6.7-bk9 kernel).

(I assume you read bundled and associated kernel and kernel header bundling, which is also bad for other reasons ;)

Hi,

You don't want to shove all headers in one rpm, as this will again make it harder to build custon kernels with their custom kernel-headers/devel packages.

Why is this ? Don't really see why this would make it harder.

Thomas

Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> And every time she sneezes I think it's love and oh lord I'm not ready for this sort of thing <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

On Sun, 2004-07-04 at 15:25, Axel Thimm wrote:

I think the benefits far outweigh the drawbacks.

Me too.

Axel, do you already have an implementation of this? What do you currently build your module packages against? Links welcome.

I peeked into the nvidia and saa7134 srpms but they do not seem to buildrequire anything special, just /usr/bin/gcc apparently through the %kmdl_parentdependencies macro. Where do all those mysterious %kmdl* macros come from?

On Sun, 2004-07-04 at 18:04, Thorsten Leemhuis wrote:

But couldn't that solved if the kernel package itself always Requires the install of the matching kernel-header-`uname -r` package?

Sort of (although ugly IMO), but as Axel said:

• you don't need the headers for running the kernel

On 2004.06.25 12:04, Jeff Spaleta wrote:

Can we please continue to focus discussion on identifying and proposing solutions for real problems associated with dropping kernel-sourcecode. So far I only see one real problem ( potentially pre-existing problem that people have grown accustom to hacking around a certain way). And that problem is simply,

'What is the best way to build add-on modules for multiple arches of the same binary kernel version with the goal of providing repository access for other people to those kernel modules packages`

Right. But this needs to include a convenient way to build modules that are present in the kernel source that Arjan doen't build (for reasons I don't understand but drive me crazy) such as ide-scsi.

Regards, Willem Riede.

On 2004.06.26 10:16, Alan Cox wrote:

On Sat, Jun 26, 2004 at 08:56:26AM -0400, Willem Riede wrote:

...

Right. But this needs to include a convenient way to build modules that are present in the kernel source that Arjan doen't build (for reasons I don't understand but drive me crazy) such as ide-scsi.

The easiest way is to edit the .config files in the source rpm package add "wr" or "ac" or whatever to the end of the version and rebuild and install.

Your extra modules are then rpm managed, and the version means that a new RH security update wil replace your kernel

Right. And then I get to do the edit all over again... I'm off to think about a way to mechanize that...

Regards, Willem Riede.

On Jun 25, 2004, Jack Neely jjneely@pams.ncsu.edu wrote:

I do like the idea of having /lib/modules/`uname -r`/build symlink point to /blah/blah/kernel-headers/arch.

This doesn't solve the problem of being able to install both i586 and i686 kernels on the same box, unless you make `uname -r` print different strings for them. Otherwise, the rpms would still conflict, just like they do today.

On Jun 26, 2004, Jack Neely jjneely@pams.ncsu.edu wrote:

Why do you need to do this? The idea is that you can build kernel modules for any kernel no matter what kernel you happen to be booted into.

Right. But you have to be able to install both kernel /build trees, since they (may) differ between i586 and i686, and then know where to locate them.

In fact you've not been able to do with with the 2.4 kernels in older versions of FC/RHL.

Yup. It's a very long-standing problem.

...

Personally, I think that having a kernel-devel package that puts the headers somewhere with both the `uname -r` and arch as part of the path would be useful.

This all sounds very much like looking for a solution after the fact.

Set aside the technical issues this "accomplished facts" policy does not speak very well for the willingness to involve the community in decision making, or even involving the community in discussing technical issues and notifying possible problems before a change is made. This rather fundamental change has not even been pre announced. This has nothing to do with the way a community project should work.

I have a tendency to agree here. There are a lot of community members that care a lot, and are far more anal about issues like this, than the people working at Red Hat which (and I respect their work) sometimes have to figure out quick and dirty hacks to problems and don't really think about it well enough to come up with something that doesn't give us all headaches after the fact.

It only needs a little bit of discussion *before* a huge sweeping change is made ,and that little bit of discussion is *exactly* what Fedora should be doing. If it can't manage that, it's pretty much doomed as a project, isn't it ?

Case in point - us packagers would have suggested a system where the kernel installs in a unique location so that multiple kernels were parallel-installable.

Thomas

Dave/Dina : future TV today ! - http://www.davedina.org/ <-*- thomas (dot) apestaart (dot) org -*-> If I could talk I'd tell you If I could smile I'd let you know You are far and away my most imaginary friend <-*- thomas (at) apestaart (dot) org -*-> URGent, best radio on the net - 24/7 ! - http://urgent.fm/

On Thu, 2004-06-24 at 02:25 +0200, Thomas Vander Stichele wrote:

...

While I'm far from an expert, the headers needed to build 3rd party modules are now packaged with the kernel proper (that's one reason it's taking so long to install a new kernel nowadays).

Looks to me like the main reason it's taking so long is because kernel post scripts run "hardlink" which is not documented and has no help output. But judging by the name, the arguments it takes, and where it's run, I'd bet good money on conjecturing that it is a script that given a path or set of paths figures out all the files are the same, then hardlinks one to the other, thus saving space.

Correct

Personally I don't think the benefit of saving space outweighs the tremendous amount of time this takes, and I can't imagine it having very much benefit if the kernel version and release of the files being evualuated aren't exactly the same. Just my idea though.

Actually, there's quite a bit of savings. I don't remember the numbers, but they were fairly absurdly high. The run of hardlink could actually be sped up if it were modified to only do the md5 checks if the file's basename is the same. I just haven't had the 'round 'tuits to look into making the change myself.

Jeremy

On Thu, 2004-06-24 at 04:55, Stephen Smoogen wrote:

On Wed, 23 Jun 2004, Jeremy Katz and others wrote about the loss of kernel-source(code)

I think the main problem is that a LOT of people use the current method

well I agree that there are quite a few people who like to build their own kernel config but based on our source tree. However the plan is to include quite a bit of information for those people on how to get the same result. It saves shipping 40Mb of redundant rpm and it saves quite a bit of installation time though.

and expect it to be there for whatever arcane method. Just removing it without openly discussing it first smacks less of a community project

we're having the discussion now right?

...

...

and expect it to be there for whatever arcane method. Just removing it without openly discussing it first smacks less of a community project

we're having the discussion now right?

after the fact.

Not exactly likely to go back to the way it was, is it?

rawhide is for experimenting and for solliciting comments and see if and why people care. It's 10x easier to discuss a change when the result is visible instead of theoretical discussions about something that sort of kinda will be like this.

On Thu, 2004-06-24 at 08:32, seth vidal wrote:

...

rawhide is for experimenting and for solliciting comments and see if and why people care. It's 10x easier to discuss a change when the result is visible instead of theoretical discussions about something that sort of kinda will be like this.

But like I said, these have already been released. They're not just in rawhide.

ehh? the removal of kernel-sourcecode is NOT in released updates.

On Thu, 24 Jun 2004 21:26:40 +0200, Ronny Buchmann ronny-vlug@vlugnet.org wrote:

But the removal of DVB modules and the name change to kernel-sourcecode are.

Let's try to keep this specific thread on topic please. Changes that have come before with regard to released-update kernel packages will only serve to muddy the specific issues associated with dropping kernel-source(code) completely as a binary package moving forward. Please lets not drag other questions into the discussion.

As a quick summary for those keeping score there are appear to be two directly relevant issues that are going to affect module builders like 3rd party public repositories and private intranet repository builders. Common end-user(and I dare say power-user) usage scenarios shouldn't be affected assuming of course that post-install module building scripts are doing the right thing and looking in /lib/modules/`uname -r`/build/ :

1) how to deal with building modules for multiple versions of the kernel just using the binary kernel packages without resorting to some pedantic games to trick rpm into letting you install multiple versions of a kernel without overwriting files for the running kernel. Ex: building modules for i586 while running i686 kernel of the same release. I personally would say encoding the i586 or i686 in the release string as a pedantic game. If the general theme here is to make kernel packages conform to how other packages are handled inside Core, encoding the arch into the version seems to go against that goal. Or do we want to do similar games with openssl and glib. A clever way to produce a kernel-devel package like other packages get would seem to conform best to the goal of treating kernel like every other package in the distro.

2)Some issues with the efficiency of how hardlink is used when installing the kernel rpms now. This issue seems fundamentally an optimization issue really, and not something that will break how 3rd party and intranet repo maintainers build kernel modules. Annoy them surely, but i think 1) is the issue that is going to have to be solved AND documented with huge blinking neon signs, skywriters and possibly a document projected onto the surface of the moon for all to read.

-jef"filing a bug to introduce an openssl-source(code) noarch package"spaleta

On Thu, Jun 24, 2004 at 08:29:28AM +0200, Arjan van de Ven wrote:

...

...

...

and expect it to be there for whatever arcane method. Just removing it without openly discussing it first smacks less of a community project

we're having the discussion now right?

after the fact.

Not exactly likely to go back to the way it was, is it?

rawhide is for experimenting and for solliciting comments and see if and why people care. It's 10x easier to discuss a change when the result is visible instead of theoretical discussions about something that sort of kinda will be like this.

So, what does your social experiment yield, obviously some people do care? Will kernel-source(code) return to remain until we find a true solution [*]?

[*] like a kernel-headers subpackage soving headers into /usr/src/kernel-headers/2.6.7-1.499-i686/ and having /lib/modules/2.6.7-1.499/build be a symlink to there? kernel module builders are happy because they can build against cold kernels (and a simple BuildRequires: kernel-headers will be enough!), users will still have the symlink.

Even if this or a similar solution will be sanctionized as the best idea since qubically sliced bread, kernel-source(code) is needed for a migration phase, so please put it back!

Am Fr, den 25.06.2004 schrieb Alexandre Oliva um 5:53:

On Jun 24, 2004, Axel Thimm Axel.Thimm@ATrpms.net wrote:

...

kernel-source(code) is needed for a migration phase

Err... You shouldn't need kernel-source to build kernel modules.

You need it in rare cases -- if you want to build modules from the kernel that were not build by rh/fedora. At the moment I'm using it to build the ntfs ;-) and the dvb modules that are missing in the released binary kernel...

IIRC the old (~in the 2.4.19 days) nforce drivers also needed the source for their audio-drivers. They patched some files from kernel-source and compiled them again. And didn't the ati radeon/fglrx drivers use the agpgart Source from the Kernel and patched them with some updates and compiled them anew? But both examples are only a "IIRC" and I may be wrong in both...

CU thl

Am Fr, den 25.06.2004 schrieb Alexandre Oliva um 8:58:

On Jun 25, 2004, Thorsten Leemhuis fedora@leemhuis.info wrote:

...

You need it in rare cases -- if you want to build modules from the kernel that were not build by rh/fedora.

If you're looking for the sources of any modules, go get them from the source rpm/tarball where they are provided. If it's not part of the kernel, you unpack the sources and build them. Why should a module that is part of the core kernel, but wasn't enabled, be any different?

Normally a simple

BuildRequire: kernel-sourcecode

was enough to have all needed Source-Code available. I don't think that is possible with SRPMS, or?

If kernel-source is missing I need do add some (AFAICS large) parts of the kernel-source package to each package im creating myself (e.g. for an NTFS or DVB-Modules RPM-Package someone might create ;-) ). Thats a large overhead there.

Therefor I'd like it very much if the kernel-source(code) RPM could stay in a form like it is today in fedora-extras (or alternatives).

CU thl

On Fri, Jun 25, 2004 at 06:45:31AM +0200, Thorsten Leemhuis wrote:

...

Err... You shouldn't need kernel-source to build kernel modules.

You need it in rare cases -- if you want to build modules from the kernel that were not build by rh/fedora. At the moment I'm using it to build the ntfs ;-) and the dvb modules that are missing in the released binary kernel...

Its much cleaner for these cases to download the srpm tweak the .config and then rebuild it, that keeps all the modules rpm managed. Even if you wanted to roll extra modules for a repository they would become patch and build add ons + a new spec file target for the item itself.

compiled them again. And didn't the ati radeon/fglrx drivers use the agpgart Source from the Kernel and patched them with some updates and compiled them anew? But both examples are only a "IIRC" and I may be wrong in both...

AGPgart in 2.4 maybe - the 2.4 AGPgart was one of the few bits of code licensed dual so that could happen. I believe 2.6 AGPGart is GPL so any driver that needed to patch it would be a derivative work and GPL - which should mean the problem doesn't arise.

What specific issues does going back to the srpm have for folks like you running 3rd party repositories

On Jun 25, 2004, Axel Thimm Axel.Thimm@ATrpms.net wrote:

Please revert this, until we find a real solution, can you?

No, I can't. I'm as much of a bystander as you.

I'm very much delighted, however, that my rawhide daily rsyncs no longer have to download N+1 copies of compressed kernel sources, where N is the number of arches I keep in my local rawhide replica. 1 copy is enough, and the kernel SRPM is the right location for the kernel sources.

If you want to build a kernel, there's a well-known procedure to turn a source RPM into a binary RPM.

If you want to build an external module, follow the procedure that has been documented for, what, years?, namely, using /lib/modules/<version>/build instead of relying on /usr/src/linux

If you want to build modules from this kernel, extract the sources using the well-known procedure and build them however you like, as if they were an external module.

The procedures above have worked for years. Sure, they may be different from those you got used to, and they may very well not be as convenient for you as forcing everyone who would like to rebuild your packages to have the kernel-sourcecode package already installed, or automatically brought in by dependencies.

The only actual change in procedure by not offering kernel-sourcecode as a binary rpm is that you can no longer bring it in with a BuildRequires. This will require binary module packages to be more conscious in (i) following the documented procedures to build kernel modules, instead of ad-hoc solutions that happened to work but, since they were not documented or recommended, were prone to changes, and (ii) in the case of building modules that are part of the kernel, packaging their sources instead of relying on a redundant and hopefully obsolete packaging artifact to obtain them.

Yeah, change can be a pain. But I really believe this change is for better.

On Jun 25, 2004, Axel Thimm Axel.Thimm@ATrpms.net wrote:

You obviously missed it, but there was a change some weeks back consolidating N to 1 noarch packages

Unfortunately rsync is dumb enough to transfer the hard-linked file multiple times, and only then hard-link it. Remains to be determined whether this is because I prime my local tree by hard-linking files that exist only in the remote tree with rpms of different versions of the same package, which sometimes saves a bit of downloads.

On Jun 25, 2004, Axel Thimm Axel.Thimm@ATrpms.net wrote:

rsync does not retrieve the second and upwards copy of the same link with -H.

Yeah. My mistake was to assume -a implied -H. Grrr.

rsync script fixed.

If you are going to change this procedure for Fedora Core to something like "install the src.rpm, edit a config file and rebuild it etc," it should not change mid-release!!!!!!!!!

Nobody, and I mean nobody, suggested doing this for FC2 so far.

If RedHat is going to continue to ship default kernels on a user-oriented "community supported" linux distribution that lack crucial functionality for a large percentage of your target audience, you've got to make it clear how to enable this stuff. And I'm not saying you won't, or that great document is not in the pipes or anything like that. I'm just anticipating the FLOOD of bad reviews that a Fedora Core distribution which lacks /usr/src/linux will generate, that's all.

we already don't ship /usr/src/linux since about RHL7.1 days...

http://linux-ntfs.sourceforge.net/rpm/index.html is what I hope google will point at....

On Jun 28, 2004, Aaron Bennett aaron.bennett@olin.edu wrote:

If you are going to change this procedure for Fedora Core to something like "install the src.rpm, edit a config file and rebuild it etc,"

Maybe we should add a /usr/src/linux/README file to some other kernel rpm, with a small blurb explaining how to get the kernel sources?

On Thu, 2004-06-24 at 15:52 -0600, Stephen Smoogen wrote:

On Thu, 2004-06-24 at 00:29, Arjan van de Ven wrote:

...

...

...

...

and expect it to be there for whatever arcane method. Just removing it without openly discussing it first smacks less of a community project

we're having the discussion now right?

after the fact.

Not exactly likely to go back to the way it was, is it?

rawhide is for experimenting and for solliciting comments and see if and why people care. It's 10x easier to discuss a change when the result is visible instead of theoretical discussions about something that sort of kinda will be like this.

While doing and asking forgiveness later is the usual internal Red Hat (TM) way.. it doesnt fit into any community other than a dictatorship.. and a harsh one at that. To put it bluntly, you either want input from the community or you dont. Doing it first and then asking for discussion ends up with both sides being defensive (as in this and previous cases) and by the time it is all worked out a lot of pissed off people on both sides really dont want to work with each other anymore.

I have to agree here. Honestly the way I see it, people post intros and are allowed to submit packages for inclusion but the bottom line is that Red Hat makes ALL the decisions and has the final say over EVERYTHING. I agree with Stephen that this type of process is not really a community at all if the community really doesn't have a say at all in the short term. And frankly we're not very far along and I'm still not convinced the 'community' is really making an impact on the long-term future of Fedora either. I'm often finding myself more frustrated and beligerant than I should be now probably because before I knew I had no real say in how you run your business... which is fair enough since I know little about how your business is run. But this illusion of a community seems like just that: an illusion.

Basically you are going to lose 3-4 days 'defending' why it had to be done, when it could have taken 3-4 days to say this is what we want to do, what can we do to make it work with you. Same amount of time lost.. just not as many tempers.

Yes and what is the result... angry users and no real compromise at all. Look at the dual-boot thing. I'm sorry but all those 'whiney windows users' are right. They didn't have a say, they were told they were screwed and were offered insults and flames BY REDHAT in return.

I'm glad someone else spoke up. Again. Kudos to you Stephen even if you don't agree with a thing I said.

-sb

-- Stephen John Smoogen smoogen@lanl.gov Los Alamos National Lab CCN-5 Sched 5/40 PH: 4-0645 Ta-03 SM-1498 MailStop B255 DP 10S Los Alamos, NM 87545 -- Please, I have had too much of the stupid today. Please wait until -- tomorrow to say these things so my tolerance has refreshed.

On Thu, Jun 24, 2004 at 08:22:43AM +0200, Thorsten Leemhuis wrote:

...

we're having the discussion now right?

Then why not put the kernel-source(code) in it's old state (like delivered with FC2) in Fedora-Extras? Then we save the space in the ISOs and those who are interested are able to download it from there? Or fedora-alternatives if it opens? ;-)

I have no problem with someone doing such an rpm, but to do it automatic gives a big mess since "binary" packages from the same src.rpm would be part of different collections. That's going to be a process nightmare.

On Thu, 2004-06-24 at 08:22 +0200, Thorsten Leemhuis wrote:

Then why not put the kernel-source(code) in it's old state (like delivered with FC2) in Fedora-Extras? Then we save the space in the ISOs and those who are interested are able to download it from there? Or fedora-alternatives if it opens? ;-)

I can just see the anti-RHers out there screaming that they aren't distributing the kernel source and are thus violating the GPL. Since RH is the Microsoft of the Linux world! <snicker>

Hoi Arjan,

we're having the discussion now right?

"Voldongen feiten politiek" that's called. First make a change and then discussing it makes it harder to get it reverted if that turned out to be a better solution (I'm not saying it is).

Also it seems we are now discussing solutions for problems created by this change after the fact instead of before. Wrong approach IMNSHO.

Leonard.

Arjan van de Ven wrote:

On Fri, Jun 25, 2004 at 02:39:13PM +0200, Leonard den Ottolander wrote:

...

Hoi Arjan,

...

we're having the discussion now right?

"Voldongen feiten politiek" that's called. First make a change and then discussing it makes it harder to get it reverted if that turned out to be a better solution (I'm not saying it is).

eh please be real. It's one bloody %define that needs toggled after I created and tested the spec infrastructure in rawhide. That is not making it harder to discuss that is making it possible to discuss. You can throw a lot of shit at me if you want, but by overstating claims that much do you really think I still enjoy putting the time and effort into making this a toggle to get that to happen? Can you explain to me why I should even bother to go the more-work route if no matter what I do I get shit thrown at me ?

I'm sorry I'm picking up a little late in this thread, but from what I gather everyone is bitching about the lost functionality that stems from there being no more kernel-source package. What on earth is wrong with installing <kernel-version>.src.rpm, and doing an:

rpmbuild -bp kernel.spec

It gets you the same source tree that you otherwise have had, and lets you build all the modules you could ever want.

Neil

Thomas Vander Stichele wrote:

Hi,

...

While I'm far from an expert, the headers needed to build 3rd party modules are now packaged with the kernel proper (that's one reason it's taking so long to install a new kernel nowadays).

Looks to me like the main reason it's taking so long is because kernel post scripts run "hardlink" which is not documented and has no help output. But judging by the name, the arguments it takes, and where it's run, I'd bet good money on conjecturing that it is a script that given a path or set of paths figures out all the files are the same, then hardlinks one to the other, thus saving space.

Personally I don't think the benefit of saving space outweighs the tremendous amount of time this takes, and I can't imagine it having very much benefit if the kernel version and release of the files being evualuated aren't exactly the same. Just my idea though.

I discussed this with Arjan during FC2 test development because I was very annoyed by the amount of time needed for hardlink. I wondered if we could run hardlink with "&" so that it runs in the background, allowing rpm to finish the transaction much sooner. I also asked if hardlink can be run as a low priority background maintenance job like prelink. Both were rejected because there is no guarantee that hardlink will be finished when rpm touches it again, and some difficulty related to free disk space calculation.

Now I think that removing hardlink from rpm %post and running it like prelink would be the best solution, but I am perhaps forgetting some past discussed details.

Warren Togami wtogami@redhat.com