Hello,
I have thought about rpm packages containing kernel modules independently of the threads in this mailing list, which Matt Domsch was friendly enough to pointed me at --- thanks, Matt. Maybe we can share some thoughts, and even move our implementations a little closer together.
We've been looking at DKMS, and decided not to use it in favor of a purely rpm based solution, so that the rpm database knows which modules belong to which packages, etc. That's the same as with your proposals on http://www.fedoraproject.org/wiki/Extras/KernelModuleProposal.
One of the benefits of DKMS we saw was the ability to query which driver packages were installed for a specific kernel. The same can be achieved by checking which packages beside the kernel package itself install *.ko files below /lib/modules/$(uname -r), so we'll achieve this feature by a script that does that. Easy enough.
You seem to also prefix kernel module package names with ``kernel-module-'' to make them easier to identify. Not really necessary I believe, especially since they can as well be listed with --whatrequires.
We thought it useful to include a unique provider prefix in the package name though, so that different vendors won't produce name clashes. Our plan was to use the LANANA provider names registry (http://www.lanana.org/) for that.
The driver name, driver version, and kernel release ($KERNELRELEASE) are also stored differently in rpm tags: our build system likes to be able to freely assign the package release number, so we don't store extra information there. Rather, we put the driver version in the Name, and the kernel release in the Version:
%define provider_prefix novell %define driver_name foo %define driver_version 1.1 %define flavor smp
%define kver %(rpm -q --qf '%{VERSION}-%{RELEASE}' kernel-source)
Name: %provider_prefix-%driver_name-%driver_version Version: %(echo %kver-%flavor | tr - _) Release: <arbitrary> Requires: kernel-%flavor = %kver
The kernel version and release %kver is determined by the installed kernel-source package.
The %flavor thing deserves some explanation: for each architecture, we have a number of differerent kernel configurations which we call "flavors". The config files for those flavors are found in arch/$ARCH/defconfig.$FLAVOR in the SUSE kernel-source package, by the way. The kernel packages are called kernel-$flavor, with version and release %kver. They additionally provide the symbol kernel = %kver. There is some documentation at http://www.suse.de/~agruen/kernel-doc/, in case anybody is interested.
Storing the driver version in the package name allows driver vendors to offer multiple versions of the same driver for use with the same kernel. These packages won't interfere with each other on updates, unless one of the drivers Provides and Obsoletes another. We have learned with pain that vendors sometimes drop features in drivers with higher versions, leading to interesting customer situations. Because of that, we want kernel updates to update driver versions only if explicitly requested.
One feature we allow is, we allow modules for multiple flavors (from the same architecture) in the same kernel driver package. For example, a kernel driver package could contain modules for 2.6.13-rc6-1-default, 2.6.13-rc6-1-smp, and 2.6.13-rc6-1-xen. It is very easy to build those modules, and most modules are really small, so this reduces the number of packages we need to build, and manage. In that case, the Version and Requires tags change to:
Version: %(echo %kver | tr - _) Requires: kernel = %kver
Another interesting case is module package reuse: often modules from the previous kernel can be reused with the new kernel after a kernel upgrade, in case a new module package is not available. Dropping driver packages might render systems unbootable (consider a raid adapter driver or similar).
It's quite easy to check if the modversions a module requires have changed. We offer to reuse module packages if possible. When a user choses to reuse a module package, we repackage the files in the old module package for the new kernel, and move modules from /lib/modules/$old_kernel to /lib/modules/$new_kernel. That way we can keep proper record in the rpm database, and everything stays nicely manageable.
What I'm also missing in your proposals is mkinitrd calls: When modules that are part of initrds get replaced, you also want to update it the initrds I guess.
Regards, Andreas.
On Thu, 2005-08-18 at 19:56 +0200, Andreas Gruenbacher wrote:
Hello,
I have thought about rpm packages containing kernel modules independently of the threads in this mailing list, which Matt Domsch was friendly enough to pointed me at --- thanks, Matt. Maybe we can share some thoughts, and even move our implementations a little closer together.
Welcome to the unified flame-war... i mean, party. :)
You seem to also prefix kernel module package names with ``kernel-module-'' to make them easier to identify. Not really necessary I believe, especially since they can as well be listed with --whatrequires.
We're doing this for user sanity as well, and to help differentiate userspace packages from kernel-module packages. (Yum might be checking for it as well, but I'm not the expert on that).
We thought it useful to include a unique provider prefix in the package name though, so that different vendors won't produce name clashes. Our plan was to use the LANANA provider names registry (http://www.lanana.org/) for that.
Ugh. I really don't want to cram everything and the kitchen sink into the package name. I'd rather see the package check for a SuSE/Fedora/Whatever only file on the system and use that to determine if its in the right place or not. We also have dist tags for that purpose in Fedora Extras.
The driver name, driver version, and kernel release ($KERNELRELEASE) are also stored differently in rpm tags: our build system likes to be able to freely assign the package release number, so we don't store extra information there. Rather, we put the driver version in the Name, and the kernel release in the Version:
Hmm. Again, I don't want to overload %{name}. That's not what its there for, imho.
What I'm also missing in your proposals is mkinitrd calls: When modules that are part of initrds get replaced, you also want to update it the initrds I guess.
That's not too hard, really. We can use the same /sbin/new-kernel-pkg that the Fedora kernel uses:
/sbin/new-kernel-pkg --mkinitrd --depmod --install %{KVERREL}
~spot
On 8/18/05, Tom 'spot' Callaway tcallawa@redhat.com wrote:
We're doing this for user sanity as well, and to help differentiate userspace packages from kernel-module packages. (Yum might be checking for it as well, but I'm not the expert on that).
Just FYI on my rawhide box, the current yum magic seems to be partly relying on packagename syntax and partly on provides syntax right now. Unless i've misread yum sniplet2 below.
yum sniplet2 seems to say if the packagename starts with "kernel-module-" then handle it like a kernel module and check for an upgrade situation versus an install situation by looking at whether the required kernel version is installed already (see sniplet1). sniplet2 also says to check to see if the package is in the installonly list (see sniplet1). For kernel modules to be in the installonly list by default they need to provide "kernel-modules" according to sniplet0
from /usr/lib/python2.4/site-packages/yum/config.py <sniplet0/> ('installonlypkgs', ['kernel', 'kernel-bigmem', 'kernel-enterprise','kernel-smp', 'kernel-modules', 'kernel-debug', 'kernel-unsupported', 'kernel-source', 'kernel-devel']), </sniplet0>
Anything providing "kernel-modules" will be treated as installonly by default.
From /usr/lib/python2.4/site-packages/yum/depsolve.py on my rawhide box
<sniplet1/> def allowedMultipleInstalls(self, po): """takes a packageObject, returns 1 or 0 depending on if the package should/can be installed multiple times with different vers like kernels and kernel modules, for example"""
if po.name in self.conf.installonlypkgs: return 1
provides = po.getProvidesNames() if filter (lambda prov: prov in self.conf.installonlypkgs, provides): return 1
return 0
def handleKernelModule(self, txmbr): """Figure out what special magic needs to be done to install/upgrade this kernel module."""
def getKernelReqs(hdr): kernels = ["kernel-%s" % a for a in rpmUtils.arch.arches.keys()] reqs = [] names = hdr[rpm.RPMTAG_REQUIRENAME] flags = hdr[rpm.RPMTAG_REQUIREFLAGS] ver = hdr[rpm.RPMTAG_REQUIREVERSION] if names is not None: reqs = zip(names, flags, ver) return filter(lambda r: r[0] in kernels, reqs)
kernelReqs = getKernelReqs(txmbr.po.returnLocalHeader()) instPkgs = self.rpmdb.returnTupleByKeyword(name=txmbr.po.name) for pkg in instPkgs: hdr = self.rpmdb.returnHeaderByTuple(pkg)[0] instKernelReqs = getKernelReqs(hdr)
for r in kernelReqs: if r in instKernelReqs: # we know that an incoming kernel module requires the # same kernel as an already installed module of the # same name. "Upgrade" this module instead of install po = packages.YumInstalledPackage(hdr) self.tsInfo.addErase(po) self.log(4, 'Removing kernel module %s upgraded to %s' % (po, txmbr.po)) break </sniplet1>
<sniplet2/> if txmbr.ts_state == 'u': if txmbr.po.name.startswith("kernel-module-"): self.handleKernelModule(txmbr) if self.allowedMultipleInstalls(txmbr.po): self.log(5, '%s converted to install' % (txmbr.po)) txmbr.ts_state = 'i' txmbr.output_state = 'installing' </sniplet2>
-jef
Well...that's one way to summon me...post my code all over the internet. :-)
The ideas and methods we had on the table a while back were...not always inline with each other. The information I had when I wrote that mess was that a kernel module package was deemed a kernel module package by having the %{name} kernel-module-<foo>. Providing kernel-modules seemed to have gotten lost. Then I became frustrated because folks kept making the specs more and more complex. Way too complex for sane package managers. Mostly the ways to get the packages not to generate source packages on each rebuild and crap.
Secondly, in the Yum world we decided to start doing some big changes and I commited this bit. Then we thought better and decided to release 2.4.0 and do the big changes in 2.5.x. Hence, I should probably remove that code from the 2.4.x series.
If we want kernel module packages to be successful they can't be that complex for the packager. Are there lots of special cases? You bet. Let the special cases be handled once in Yum code. Let the other special cases be handled once in Plague. Otherwise, each packager will do things ever so slightly different which leads to breakage. Not to mention how tired we will be of correcting and explaining.
Jack
On Thu, 2005-08-18 at 16:55 -0400, Jeff Spaleta wrote:
On 8/18/05, Tom 'spot' Callaway tcallawa@redhat.com wrote:
We're doing this for user sanity as well, and to help differentiate userspace packages from kernel-module packages. (Yum might be checking for it as well, but I'm not the expert on that).
Just FYI on my rawhide box, the current yum magic seems to be partly relying on packagename syntax and partly on provides syntax right now. Unless i've misread yum sniplet2 below.
yum sniplet2 seems to say if the packagename starts with "kernel-module-" then handle it like a kernel module and check for an upgrade situation versus an install situation by looking at whether the required kernel version is installed already (see sniplet1). sniplet2 also says to check to see if the package is in the installonly list (see sniplet1). For kernel modules to be in the installonly list by default they need to provide "kernel-modules" according to sniplet0
from /usr/lib/python2.4/site-packages/yum/config.py
<sniplet0/> ('installonlypkgs', ['kernel', 'kernel-bigmem', 'kernel-enterprise','kernel-smp', 'kernel-modules', 'kernel-debug', 'kernel-unsupported', 'kernel-source', 'kernel-devel']), </sniplet0>
Anything providing "kernel-modules" will be treated as installonly by default.
From /usr/lib/python2.4/site-packages/yum/depsolve.py on my rawhide box
<sniplet1/> def allowedMultipleInstalls(self, po): """takes a packageObject, returns 1 or 0 depending on if the package should/can be installed multiple times with different vers like kernels and kernel modules, for example"""
if po.name in self.conf.installonlypkgs: return 1 provides = po.getProvidesNames() if filter (lambda prov: prov in self.conf.installonlypkgs, provides): return 1 return 0
def handleKernelModule(self, txmbr): """Figure out what special magic needs to be done to install/upgrade this kernel module."""
def getKernelReqs(hdr): kernels = ["kernel-%s" % a for a in rpmUtils.arch.arches.keys()] reqs = [] names = hdr[rpm.RPMTAG_REQUIRENAME] flags = hdr[rpm.RPMTAG_REQUIREFLAGS] ver = hdr[rpm.RPMTAG_REQUIREVERSION] if names is not None: reqs = zip(names, flags, ver) return filter(lambda r: r[0] in kernels, reqs) kernelReqs = getKernelReqs(txmbr.po.returnLocalHeader()) instPkgs = self.rpmdb.returnTupleByKeyword(name=txmbr.po.name) for pkg in instPkgs: hdr = self.rpmdb.returnHeaderByTuple(pkg)[0] instKernelReqs = getKernelReqs(hdr) for r in kernelReqs: if r in instKernelReqs: # we know that an incoming kernel module requires the # same kernel as an already installed module of the # same name. "Upgrade" this module instead of install po = packages.YumInstalledPackage(hdr) self.tsInfo.addErase(po) self.log(4, 'Removing kernel module %s upgraded to %s' % (po, txmbr.po)) break
</sniplet1>
<sniplet2/> if txmbr.ts_state == 'u': if txmbr.po.name.startswith("kernel-module-"): self.handleKernelModule(txmbr) if self.allowedMultipleInstalls(txmbr.po): self.log(5, '%s converted to install' % (txmbr.po)) txmbr.ts_state = 'i' txmbr.output_state = 'installing' </sniplet2>
-jef
-- Fedora-packaging mailing list Fedora-packaging@redhat.com https://www.redhat.com/mailman/listinfo/fedora-packaging
On 8/18/05, Jack Neely jjneely@pams.ncsu.edu wrote:
If we want kernel module packages to be successful they can't be that complex for the packager. Are there lots of special cases? You bet. Let the special cases be handled once in Yum code. Let the other special cases be handled once in Plague. Otherwise, each packager will do things ever so slightly different which leads to breakage. Not to mention how tired we will be of correcting and explaining.
I'm just trying to get this conversation moving forward again by making sure everyone is aware of where the current implementation stands. This particular issue seems to be forever stalled in a debate over where to start.. policy or tools. I'd hate to see this conversation frustrated further by an inaccurate accessment of current reality. If packagers provided "kernel-modules" the current magic in yum would work pretty well for most of the situations an end-user would care about. What are the real hold ups here... why isn't it good enough to make a policy decision that all Core and Extras module packages provide "kernel-modules" and be named "kernel-module-whatever" as a starting point to get some packages out?
-jef
I'm just trying to get this conversation moving forward again by making sure everyone is aware of where the current implementation stands. This particular issue seems to be forever stalled in a debate over where to start.. policy or tools. I'd hate to see this
Well, we've got a small bit of the functionality in Yum working...if oddly. I don't mind doing that work but we need some idea of policy that I can code to. :-)
conversation frustrated further by an inaccurate accessment of current reality. If packagers provided "kernel-modules" the current magic in yum would work pretty well for most of the situations an end-user would care about. What are the real hold ups here... why isn't it good enough to make a policy decision that all Core and Extras module packages provide "kernel-modules" and be named "kernel-module-whatever" as a starting point to get some packages out?
How I would like things to be:
Kernel module packages MUST provide "kernel-modules". Not "kernel-module". This keeps in line with previous work and already existing code. This provides will identify to Yum/Plague/Whatever that this package is a kernel module and needs special treatment.
Kernel module packages MUST require kernel-%{_targetcpu} = ${kver}. This identifies to Yum what kernel this module works with and will identify other kernel module packages that need to be removed.
The release tag will be <RELEASE>.%{?kver:.%(echo %{kver} | tr - _)} or something along those lines. This provides no information to Yum/Plague but gives the package a unique EVR. Its really hard to correctly parse information like a kernel version out of the release tag.
I really don't like the kernel-module-foo-source base package that produces a subpackage of kernel-module-foo. This adds way to much complexity. The goal here, IIRC, is to be able to have one src.rpm for all the times that the kernel-module-foo has been rebuilt for each kernel version/flavor. Being that the release tag already contains a lot of our magic I suggest we make the release tag be the following.
Release: [release]%{!?sourcepackage:%(echo .%{kver} | tr - _)}
where [release] is the packager's release number of the package.
With this idea we ignore/delete all srpms that are created with rebuilds of the kerenl-module-foo package. To generate the official srpm the build system does:
rpmbuild -bs kernel-module-foo.spec [defines] \ --define "sourcepackage 1" [targets]
If this srpm already exists in the repository then ignore it. Otherwise, you have a new version of the kernel module itself rather than just 15 hundred rebuilds of it.
Jack
-jef
-- Fedora-packaging mailing list Fedora-packaging@redhat.com https://www.redhat.com/mailman/listinfo/fedora-packaging
On Thursday 18 August 2005 21:27, Tom 'spot' Callaway wrote:
On Thu, 2005-08-18 at 19:56 +0200, Andreas Gruenbacher wrote:
We thought it useful to include a unique provider prefix in the package name though, so that different vendors won't produce name clashes. Our plan was to use the LANANA provider names registry (http://www.lanana.org/) for that.
Ugh. I really don't want to cram everything and the kitchen sink into the package name. I'd rather see the package check for a SuSE/Fedora/Whatever only file on the system and use that to determine if its in the right place or not. We also have dist tags for that purpose in Fedora Extras.
My example was not perfectly well chosen. The idea was to have provider prefixes like adaptec, nvidia, ati and similar, so an example out-of-line aic7xxx upgrade would be adaptec-aic7xxx-8.9.10-2.6.13_99_smp-3, etc.
The driver name, driver version, and kernel release ($KERNELRELEASE) are also stored differently in rpm tags: our build system likes to be able to freely assign the package release number, so we don't store extra information there. Rather, we put the driver version in the Name, and the kernel release in the Version:
Hmm. Again, I don't want to overload %{name}. That's not what its there for, imho.
What makes %version or %release more appropriate for overloading? With the driver version as part of %version or %release, you can't easily offer packages for more than one version of a driver for the same kernel, and yet have working updates. I would be surprised if you didn't ever have this situation with RHEL.
-- Andreas.
On Fri, 2005-08-19 at 00:16 +0200, Andreas Gruenbacher wrote:
My example was not perfectly well chosen. The idea was to have provider prefixes like adaptec, nvidia, ati and similar, so an example out-of-line aic7xxx upgrade would be adaptec-aic7xxx-8.9.10-2.6.13_99_smp-3, etc.
I don't see it serving any purpose. I don't want to have a field that people are trying to shoe-horning ugliness into. This just opens the door to rpms with horrible provider prefixes, and rpms with %{name}:
InternationalBusinessMachinesJapanCOPYRIGHT2005ALLRIGHTSRESERVED-mwave-3.1415972-2.6.13_99_smp
... when all we really need is:
kernel-module-mwave
The driver name, driver version, and kernel release ($KERNELRELEASE) are also stored differently in rpm tags: our build system likes to be able to freely assign the package release number, so we don't store extra information there. Rather, we put the driver version in the Name, and the kernel release in the Version:
Hmm. Again, I don't want to overload %{name}. That's not what its there for, imho.
What makes %version or %release more appropriate for overloading? With the driver version as part of %version or %release, you can't easily offer packages for more than one version of a driver for the same kernel, and yet have working updates. I would be surprised if you didn't ever have this situation with RHEL.
Users won't expect to have the name be overloaded. Users want to search for "kernel-module-foo", query the rpmdb for kernel-module-foo. Not to mention the nightmare of tracking it in bugzilla, and generating massive amounts of unnecessary SRPMs.
And we can certainly offer multiple packages.
kernel-module-foo-1.2-1.2.6.13_93smp (driver version 1.2, build 1) kernel-module-foo-1.2-2.2.6.13_93smp (driver version 1.2, build 2) kernel-moudle-foo-1.3-1.2.6.13_93smp (driver version 1.3, build 1)
~spot
On Friday 19 August 2005 01:00, Tom 'spot' Callaway wrote:
On Fri, 2005-08-19 at 00:16 +0200, Andreas Gruenbacher wrote:
My example was not perfectly well chosen. The idea was to have provider prefixes like adaptec, nvidia, ati and similar, so an example out-of-line aic7xxx upgrade would be adaptec-aic7xxx-8.9.10-2.6.13_99_smp-3, etc.
I don't see it serving any purpose. I don't want to have a field that people are trying to shoe-horning ugliness into. This just opens the door to rpms with horrible provider prefixes, and rpms with %{name}:
InternationalBusinessMachinesJapanCOPYRIGHT2005ALLRIGHTSRESERVED-mwave-3.14 15972-2.6.13_99_smp
... when all we really need is:
kernel-module-mwave
Great, thanks for the thoughtful example. I was talking about a prefix from the LANANA provider name registry, http://lanana.org/lsbreg/providers/index.html, and I'm also aware that many potential providers don't have an entry there, yet. Also see http://lsbbook.gforge.freestandards.org/lanana.html about LSB's package naming recommendations.
The two proposals, next to each other, and even for the same example, seem to be:
Name: kernel-module-aic7xxx Version: 6.2.36 Release: 1.2.6.13_rc6_1_smp
vs.
Name: adaptec-aic7xxx-6.2.36 Version: 2.6.13_rc6_1_smp Release: 1
The driver name, driver version, and kernel release ($KERNELRELEASE) are also stored differently in rpm tags: our build system likes to be able to freely assign the package release number, so we don't store extra information there. Rather, we put the driver version in the Name, and the kernel release in the Version:
Hmm. Again, I don't want to overload %{name}. That's not what its there for, imho.
What makes %version or %release more appropriate for overloading? With the driver version as part of %version or %release, you can't easily offer packages for more than one version of a driver for the same kernel, and yet have working updates. I would be surprised if you didn't ever have this situation with RHEL.
Users won't expect to have the name be overloaded. Users want to search for "kernel-module-foo", query the rpmdb for kernel-module-foo. Not to mention the nightmare of tracking it in bugzilla, and generating massive amounts of unnecessary SRPMs.
Putting aside FUD for a second, where do you see problems querying the rpm database, or in Bugzilla? And why do you think the number of source rpms would change at all?
And we can certainly offer multiple packages.
kernel-module-foo-1.2-1.2.6.13_93smp (driver version 1.2, build 1) kernel-module-foo-1.2-2.2.6.13_93smp (driver version 1.2, build 2) kernel-moudle-foo-1.3-1.2.6.13_93smp (driver version 1.3, build 1)
You can have multiple packets next to each other, but rpm --freshen (and other tools using the same logic) won't work as expected anymore: you will always end up with the most recent driver version. Sticking with the same driver version by default will break.
-- Andreas.
On Fri, 2005-08-19 at 12:06 +0200, Andreas Gruenbacher wrote:
Great, thanks for the thoughtful example. I was talking about a prefix from the LANANA provider name registry, http://lanana.org/lsbreg/providers/index.html, and I'm also aware that many potential providers don't have an entry there, yet.
You're presuming that third party entities will follow the letter of such a proposal, rather than attempting to cram garbage in (but stay in the spirit of the proposal). ACPI and DMA prove that conclusively incorrect.
Putting aside FUD for a second, where do you see problems querying the rpm database, or in Bugzilla?
What's the bugzilla entry? adaptec-aic7xxx-6.2.36? adaptec-aic7xxx-6.2.37? adaptec-aic7xxx-6.2.38?
rpm -q adaptec-aic7xxx fails. Not only does the user need to know the driver name, they also need to remember the vendor.
And why do you think the number of source rpms would change at all?
Source rpms are generated from %{name}. By putting changing and unique variables in %{name}, you generate a LOT of srpms.
And we can certainly offer multiple packages.
kernel-module-foo-1.2-1.2.6.13_93smp (driver version 1.2, build 1) kernel-module-foo-1.2-2.2.6.13_93smp (driver version 1.2, build 2) kernel-moudle-foo-1.3-1.2.6.13_93smp (driver version 1.3, build 1)
You can have multiple packets next to each other, but rpm --freshen (and other tools using the same logic) won't work as expected anymore: you will always end up with the most recent driver version. Sticking with the same driver version by default will break.
We've fixed the tools (specifically yum) to handle this condition.
~spot
packaging@lists.fedoraproject.org