There are still some things I need to fix before this is really ready, but in the spirit of release-early here is the current kernel.spec file diff I have. This does a few things.
Macros and %files:
1) Moves the kernel_reqprovconf macro down a bit and makes the plain "kernel" package a meta-package
2) All actual kernel builds are now done via the kernel_variant_package macro. If no variant is given, it builds kernel-core. Otherwise it builds kernel-<variant>-core.
3) Adds a kernel_drivers_package subpackage macro that kernel_variant_package expands and includes. Also defines the proper post and postun scriptlets.
4) Makes the kernel_variant_files macro use file lists for the kernel-core and kernel-drivers subpackages.
File list generation:
The hunk in the middle is what generates the file lists that item 4 above uses. It does
1) calls a script called "filter_modules" that goes through and removes the modules we don't consider important enough to be in kernel-core in the RPM_BUILD_DIR.
2) calls depmod on the now filtered tree, as kernel-core should be self-contained.
3) Restores the module directory from a copy we've made prior, so that RPM can do it's package assembly later.
4) Save off the file lists in the proper location for later use and cleanup.
TODO:
Right now this is what has been building in my COPR for the past week. It works well on x86_64 (which is all I've tested), and i686 builds but I know it's broken on ARM. To finish it up, I need:
1) Review from you all.
2) A per-arch filter list, because the existing one that works on x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
3) Further testing on all arches.
4) To make the depmod call fatal to the build. Without making this fatal we run the risk of pushing kernel-core packages that aren't self-contained and that seems pretty bad to me. It shouldn't be a huge issue as I would think most of the breakage is going to come from merge window stuff and settle down after that.
josh
diff --git a/kernel.spec b/kernel.spec index e055bb2..4ffa1b6 100644 --- a/kernel.spec +++ b/kernel.spec @@ -34,7 +34,7 @@ Summary: The Linux kernel # For non-released -rc kernels, this will be appended after the rcX and # gitX tags, so a 3 here would become part of release "0.rcX.gitX.3" # -%global baserelease 1 +%global baserelease 8 %global fedora_build %{baserelease}
# base_sublevel is the kernel version we're starting with and patching @@ -385,29 +385,6 @@ Summary: The Linux kernel %define kernel_prereq fileutils, systemd >= 203-2 %define initrd_prereq dracut >= 027
-# -# This macro does requires, provides, conflicts, obsoletes for a kernel package. -# %%kernel_reqprovconf <subpackage> -# It uses any kernel_<subpackage>_conflicts and kernel_<subpackage>_obsoletes -# macros defined above. -# -%define kernel_reqprovconf \ -Provides: kernel = %{rpmversion}-%{pkg_release}\ -Provides: kernel-%{_target_cpu} = %{rpmversion}-%{pkg_release}%{?1:+%{1}}\ -Provides: kernel-drm-nouveau = 16\ -Provides: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\ -Requires(pre): %{kernel_prereq}\ -Requires(pre): %{initrd_prereq}\ -Requires(pre): linux-firmware >= 20130724-29.git31f6b30\ -Requires(preun): systemd >= 200\ -%{expand:%%{?kernel%{?1:_%{1}}_conflicts:Conflicts: %%{kernel%{?1:_%{1}}_conflicts}}}\ -%{expand:%%{?kernel%{?1:_%{1}}_obsoletes:Obsoletes: %%{kernel%{?1:_%{1}}_obsoletes}}}\ -%{expand:%%{?kernel%{?1:_%{1}}_provides:Provides: %%{kernel%{?1:_%{1}}_provides}}}\ -# We can't let RPM do the dependencies automatic because it'll then pick up\ -# a correct but undesirable perl dependency from the module headers which\ -# isn't required for the kernel proper to function\ -AutoReqProv: no\ -%{nil}
Name: kernel%{?variant} Group: System Environment/Kernel @@ -419,8 +396,9 @@ Release: %{pkg_release} # SET %%nobuildarches (ABOVE) INSTEAD ExclusiveArch: noarch %{all_x86} x86_64 ppc ppc64 ppc64p7 s390 s390x %{arm} aarch64 ppc64le ExclusiveOS: Linux +Requires: kernel-%{?variant:%{variant}-}core-uname-r = %{KVERREL}%{?variant} +Requires: kernel-%{?variant:%{variant}-}drivers-uname-r = %{KVERREL}%{?variant}
-%kernel_reqprovconf
# # List the packages used during the kernel build @@ -464,6 +442,7 @@ Source15: merge.pl Source16: mod-extra.list Source17: mod-extra.sh Source18: mod-sign.sh +Source99: filter-modules.sh %define modsign_cmd %{SOURCE18}
Source19: Makefile.release @@ -649,10 +628,31 @@ Patch25044: iwlwifi-dvm-take-mutex-when-sending-SYNC-BT-config-command.patch BuildRoot: %{_tmppath}/kernel-%{KVERREL}-root
%description -The kernel package contains the Linux kernel (vmlinuz), the core of any -Linux operating system. The kernel handles the basic functions -of the operating system: memory allocation, process allocation, device -input and output, etc. +The kernel meta package + +# +# This macro does requires, provides, conflicts, obsoletes for a kernel package. +# %%kernel_reqprovconf <subpackage> +# It uses any kernel_<subpackage>_conflicts and kernel_<subpackage>_obsoletes +# macros defined above. +# +%define kernel_reqprovconf \ +Provides: kernel = %{rpmversion}-%{pkg_release}\ +Provides: kernel-%{_target_cpu} = %{rpmversion}-%{pkg_release}%{?1:+%{1}}\ +Provides: kernel-drm-nouveau = 16\ +Provides: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\ +Requires(pre): %{kernel_prereq}\ +Requires(pre): %{initrd_prereq}\ +Requires(pre): linux-firmware >= 20130724-29.git31f6b30\ +Requires(preun): systemd >= 200\ +%{expand:%%{?kernel%{?1:_%{1}}_conflicts:Conflicts: %%{kernel%{?1:_%{1}}_conflicts}}}\ +%{expand:%%{?kernel%{?1:_%{1}}_obsoletes:Obsoletes: %%{kernel%{?1:_%{1}}_obsoletes}}}\ +%{expand:%%{?kernel%{?1:_%{1}}_provides:Provides: %%{kernel%{?1:_%{1}}_provides}}}\ +# We can't let RPM do the dependencies automatic because it'll then pick up\ +# a correct but undesirable perl dependency from the module headers which\ +# isn't required for the kernel proper to function\ +AutoReqProv: no\ +%{nil}
%package headers Summary: Header files for the Linux kernel for use by glibc @@ -834,53 +834,65 @@ Provides: kernel%{?1:-%{1}}-modules-extra = %{version}-%{release}%{?1:+%{1}}\ Provides: installonlypkg(kernel-module)\ Provides: kernel%{?1:-%{1}}-modules-extra-uname-r = %{KVERREL}%{?1:+%{1}}\ Requires: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\ +Requires: kernel-drivers-uname-r = %{KVERREL}%{?1:+%{1}}\ AutoReqProv: no\ %description -n kernel%{?variant}%{?1:-%{1}}-modules-extra\ This package provides less commonly used kernel modules for the %{?2:%{2} }kernel package.\ %{nil}
# +# This macro creates a kernel-<subpackage>-drivers package. +# %%kernel_drivers_package <subpackage> <pretty-name> +# +%define kernel_drivers_package() \ +%package %{?1:%{1}-}drivers\ +Summary: kernel modules to match the %{?2:%{2}-}core kernel\ +Group: System Environment/Kernel\ +Provides: kernel%{?1:-%{1}}-drivers-%{_target_cpu} = %{version}-%{release}\ +Provides: kernel-drivers-%{_target_cpu} = %{version}-%{release}%{?1:+%{1}}\ +Provides: kernel-drivers = %{version}-%{release}%{?1:+%{1}}\ +Provides: installonlypkg(kernel-module)\ +Provides: kernel-drivers-uname-r = %{KVERREL}%{?1:+%{1}}\ +Requires: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\ +AutoReqProv: no\ +%description -n kernel%{?variant}%{?1:-%{1}}-drivers\ +This package provides commonly used kernel modules for the %{?2:%{2}-}core kernel package.\ +%{nil} + +# # This macro creates a kernel-<subpackage> and its -devel and -debuginfo too. # %%define variant_summary The Linux kernel compiled for <configuration> # %%kernel_variant_package [-n <pretty-name>] <subpackage> # %define kernel_variant_package(n:) \ -%package %1\ +%package %{?1:%{1}-}core\ Summary: %{variant_summary}\ Group: System Environment/Kernel\ -%kernel_reqprovconf\ -%{expand:%%kernel_devel_package %1 %{!?-n:%1}%{?-n:%{-n*}}}\ +Provides: kernel-%{?1:%{1}-}core-uname-r = %{KVERREL}%{?1:+%{1}}\ +%{expand:%%kernel_reqprovconf}\ +%{expand:%%kernel_devel_package %{?1:%{1}} %{!?{-n}:%{1}}%{?{-n}:%{-n*}}}\ +%{expand:%%kernel_drivers_package %{?1:%{1}} %{!?{-n}:%{1}}%{?{-n}:%{-n*}}}\ %if %{with_extra}\ -%{expand:%%kernel_modules_extra_package %1 %{!?-n:%1}%{?-n:%{-n*}}}\ +%{expand:%%kernel_modules_extra_package %{?1:%{1}} %{!?{-n}:%{1}}%{?{-n}:%{-n*}}}\ %endif\ -%{expand:%%kernel_debuginfo_package %1}\ +%{expand:%%kernel_debuginfo_package %{?1:%{1}}}\ %{nil}
- -# First the auxiliary packages of the main kernel package. -%kernel_devel_package -%if %{with_extra} -%kernel_modules_extra_package -%endif -%kernel_debuginfo_package - - # Now, each variant package.
%define variant_summary The Linux kernel compiled for SMP machines %kernel_variant_package -n SMP smp -%description smp +%description smp-core This package includes a SMP version of the Linux kernel. It is required only on machines with two or more CPUs as well as machines with hyperthreading technology.
Install the kernel-smp package if your machine uses two or more CPUs.
- %ifnarch armv7hl %define variant_summary The Linux kernel compiled for PAE capable machines %kernel_variant_package %{pae} -%description %{pae} +%description %{pae}-core This package includes a version of the Linux kernel with support for up to 64GB of high memory. It requires a CPU with Physical Address Extensions (PAE). The non-PAE kernel can only address up to 4GB of memory. @@ -888,7 +900,7 @@ Install the kernel-PAE package if your machine has more than 4GB of memory. %else %define variant_summary The Linux kernel compiled for Cortex-A15 %kernel_variant_package %{pae} -%description %{pae} +%description %{pae}-core This package includes a version of the Linux kernel with support for Cortex-A15 devices with LPAE and HW virtualisation support %endif @@ -897,7 +909,7 @@ Cortex-A15 devices with LPAE and HW virtualisation support %define variant_summary The Linux kernel compiled with extra debugging enabled for PAE capable machines %kernel_variant_package %{pae}debug Obsoletes: kernel-PAE-debug -%description %{pae}debug +%description %{pae}debug-core This package includes a version of the Linux kernel with support for up to 64GB of high memory. It requires a CPU with Physical Address Extensions (PAE). The non-PAE kernel can only address up to 4GB of memory. @@ -910,7 +922,7 @@ on kernel bugs, as some of these options impact performance noticably.
%define variant_summary The Linux kernel compiled with extra debugging enabled %kernel_variant_package debug -%description debug +%description debug-core The kernel package contains the Linux kernel (vmlinuz), the core of any Linux operating system. The kernel handles the basic functions of the operating system: memory allocation, process allocation, device @@ -920,6 +932,16 @@ This variant of the kernel has numerous debugging options enabled. It should only be installed when trying to gather additional information on kernel bugs, as some of these options impact performance noticably.
+# And finally the main -core package + +%define variant_summary The Linux kernel +%kernel_variant_package +%description core +The kernel package contains the Linux kernel (vmlinuz), the core of any +Linux operating system. The kernel handles the basic functions +of the operating system: memory allocation, process allocation, device +input and output, etc. +
%prep # do a few sanity-checks for --with *only builds @@ -1589,6 +1611,53 @@ BuildKernel() { %{SOURCE17} $RPM_BUILD_ROOT/lib/modules/$KernelVer %{SOURCE16} %endif
+ # + # Generate the kernel-core and kernel-drivers files lists + # + + # Copy the System.map file for depmod to use, and create a backup of the + # full module tree so we can restore it after we're done filtering + cp System.map $RPM_BUILD_ROOT/. + pushd $RPM_BUILD_ROOT + mkdir restore + cp -r lib/modules/$KernelVer/* restore/. + + # don't include anything going into k-m-e in the file lists + rm -rf lib/modules/$KernelVer/extra + + # Find all the module files and filter them out into the core and drivers + # lists. This actually removes anything going into -drivers from the dir. + find lib/modules/$KernelVer/kernel -name *.ko | sort -n > modules.list + %{SOURCE99} modules.list + + # Run depmod on the resulting module tree and make sure it isn't broken + depmod -b . -aeF ./System.map $KernelVer + # remove files that will be auto generated by depmod at rpm -i time + pushd $RPM_BUILD_ROOT/lib/modules/$KernelVer/ + rm -f modules.{alias*,builtin.bin,dep*,*map,symbols*,devname,softdep} + popd + + # Go back and find all of the various directories in the tree. We use this + # for the dir lists in kernel-core + find lib/modules/$KernelVer/kernel -type d | sort -n > module-dirs.list + + # Cleanup + rm System.map + cp -r restore/* lib/modules/$KernelVer/. + rm -rf restore + popd + + # Make sure the files lists start with absolute paths or rpmbuild fails. + # Also add in the dir entries + sed -e 's/^lib*//lib/' $RPM_BUILD_ROOT/k-d.list > ../kernel${Flavour:+-${Flavour}}-drivers.list + sed -e 's/^lib*/%dir /lib/' $RPM_BUILD_ROOT/module-dirs.list > ../kernel${Flavour:+-${Flavour}}-core.list + sed -e 's/^lib*//lib/' $RPM_BUILD_ROOT/modules.list >> ../kernel${Flavour:+-${Flavour}}-core.list + + # Cleanup + rm -f $RPM_BUILD_ROOT/k-d.list + rm -f $RPM_BUILD_ROOT/modules.list + rm -f $RPM_BUILD_ROOT/module-dirs.list + %if %{signmodules} # Save the signing keys so we can sign the modules in __modsign_install_post cp signing_key.priv signing_key.priv.sign${Flav} @@ -1857,11 +1926,28 @@ fi\
# # This macro defines a %%post script for a kernel*-modules-extra package. +# It also defines a %%postun script that does the same thing. # %%kernel_modules_extra_post [<subpackage>] # %define kernel_modules_extra_post() \ %{expand:%%post %{?1:%{1}-}modules-extra}\ /sbin/depmod -a %{KVERREL}%{?1:+%{1}}\ +%{nil}\ +%{expand:%%postun %{?1:%{1}-}modules-extra}\ +/sbin/depmod -a %{KVERREL}%{?1:+%{1}}\ +%{nil} + +# +# This macro defines a %%post script for a kernel*-drivers package. +# It also defines a %%postun script that does the same thing. +# %%kernel_drivers_post [<subpackage>] +# +%define kernel_drivers_post() \ +%{expand:%%post %{?1:%{1}-}drivers}\ +/sbin/depmod -a %{KVERREL}%{?1:+%{1}}\ +%{nil}\ +%{expand:%%postun %{?1:%{1}-}drivers}\ +/sbin/depmod -a %{KVERREL}%{?1:+%{1}}\ %{nil}
# This macro defines a %%posttrans script for a kernel package. @@ -1869,7 +1955,7 @@ fi\ # More text can follow to go at the end of this variant's %%post. # %define kernel_variant_posttrans() \ -%{expand:%%posttrans %{?1}}\ +%{expand:%%posttrans %{?1:%{1}-}core}\ /bin/kernel-install add %{KVERREL}%{?1:+%{1}} /%{image_install_path}/vmlinuz-%{KVERREL}%{?1:+%{1}} || exit $?\ %{nil}
@@ -1880,11 +1966,12 @@ fi\ # %define kernel_variant_post(v:r:) \ %{expand:%%kernel_devel_post %{?-v*}}\ +%{expand:%%kernel_drivers_post %{?-v*}}\ %if %{with_extra}\ %{expand:%%kernel_modules_extra_post %{?-v*}}\ %endif\ %{expand:%%kernel_variant_posttrans %{?-v*}}\ -%{expand:%%post %{?-v*}}\ +%{expand:%%post %{?-v*:%{-v*}-}core}\ %{-r:\ if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ] &&\ [ -f /etc/sysconfig/kernel ]; then\ @@ -1897,7 +1984,7 @@ fi}\ # %%kernel_variant_preun <subpackage> # %define kernel_variant_preun() \ -%{expand:%%preun %{?1}}\ +%{expand:%%preun %{?1:%{1}-}core}\ /bin/kernel-install remove %{KVERREL}%{?1:+%{1}} /%{image_install_path}/vmlinuz-%{KVERREL}%{?1:+%{1}} || exit $?\ %{nil}
@@ -1998,6 +2085,10 @@ fi %endif %endif # with_perf
+# empty meta-package +%files +%defattr(-,root,root) + # This is %%{image_install_path} on an arch where that includes ELF files, # or empty otherwise. %define elf_image_install_path %{?kernel_image_elf:%{image_install_path}} @@ -2009,7 +2100,7 @@ fi # %define kernel_variant_files(k:) \ %if %{1}\ -%{expand:%%files %{?2}}\ +%{expand:%%files -f kernel-%{?2:%{2}-}core.list %{?2:%{2}-}core}\ %defattr(-,root,root)\ /%{image_install_path}/%{?-k:%{-k*}}%{!?-k:vmlinuz}-%{KVERREL}%{?2:+%{2}}\ /%{image_install_path}/.vmlinuz-%{KVERREL}%{?2:+%{2}}.hmac \ @@ -2018,9 +2109,10 @@ fi %endif\ %attr(600,root,root) /boot/System.map-%{KVERREL}%{?2:+%{2}}\ /boot/config-%{KVERREL}%{?2:+%{2}}\ +%ghost /boot/initramfs-%{KVERREL}%{?2:+%{2}}.img\ %dir /lib/modules\ %dir /lib/modules/%{KVERREL}%{?2:+%{2}}\ -/lib/modules/%{KVERREL}%{?2:+%{2}}/kernel\ +%dir /lib/modules/%{KVERREL}%{?2:+%{2}}/kernel\ /lib/modules/%{KVERREL}%{?2:+%{2}}/build\ /lib/modules/%{KVERREL}%{?2:+%{2}}/source\ /lib/modules/%{KVERREL}%{?2:+%{2}}/updates\ @@ -2029,7 +2121,8 @@ fi /etc/ld.so.conf.d/kernel-%{KVERREL}%{?2:+%{2}}.conf\ %endif\ /lib/modules/%{KVERREL}%{?2:+%{2}}/modules.*\ -%ghost /boot/initramfs-%{KVERREL}%{?2:+%{2}}.img\ +%{expand:%%files -f kernel-%{?2:%{2}-}drivers.list %{?2:%{2}-}drivers}\ +%defattr(-,root,root)\ %{expand:%%files %{?2:%{2}-}devel}\ %defattr(-,root,root)\ /usr/src/kernels/%{KVERREL}%{?2:+%{2}}\
On Mon, Mar 17, 2014 at 03:13:54PM -0400, Josh Boyer wrote:
TODO:
Right now this is what has been building in my COPR for the past week. It works well on x86_64 (which is all I've tested), and i686 builds but I know it's broken on ARM. To finish it up, I need:
- Review from you all.
Ahem.
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops.
- Further testing on all arches.
http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=1711902 http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=2255745 http://koji.fedoraproject.org/koji/taskinfo?taskID=6646945
The last link for i686 wasn't really needed as both i686 and x86_64 are covered by the COPR builds. I did it anyway because it was easy to kick off.
- To make the depmod call fatal to the build. Without making this
fatal we run the risk of pushing kernel-core packages that aren't self-contained and that seems pretty bad to me. It shouldn't be a huge issue as I would think most of the breakage is going to come from merge window stuff and settle down after that.
After I get s390x sorted I'll probably start making depmod fail the build locally.
josh
Josh Boyer (jwboyer@fedoraproject.org) said:
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops.
Is this even needed on s390 for reasons other than consistency? Similarly with power, is the idea to have a core kernel for running on an LPAR and then -drivers for the rest of it?
Bill
On Wed, Mar 19, 2014 at 2:02 PM, Bill Nottingham notting@splat.cc wrote:
Josh Boyer (jwboyer@fedoraproject.org) said:
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops.
Is this even needed on s390 for reasons other than consistency? Similarly with power, is the idea to have a core kernel for running on an LPAR and then -drivers for the rest of it?
Needed? Probably not. At the moment it's not possible to build a normal kernel on one arch and the split on another. If we're going to go off and make changes to anaconda and yum and dnf to cope with this, consistency on what is shipped is probably a good thing.
That being said, it is flexible in terms of the content of those packages. So ppc64 could do what you suggest. s390x would arguably just shove almost everything in -drivers. In reality, I expect most arches to just install both packages anyway.
josh
On Wednesday, March 19, 2014, 2:09:15 PM, Josh Boyer wrote:
On Wed, Mar 19, 2014 at 2:02 PM, Bill Nottingham notting@splat.cc wrote:
Josh Boyer (jwboyer@fedoraproject.org) said:
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops.
Is this even needed on s390 for reasons other than consistency? Similarly with power, is the idea to have a core kernel for running on an LPAR and then -drivers for the rest of it?
Needed? Probably not. At the moment it's not possible to build a normal kernel on one arch and the split on another. If we're going to go off and make changes to anaconda and yum and dnf to cope with this, consistency on what is shipped is probably a good thing.
That being said, it is flexible in terms of the content of those packages. So ppc64 could do what you suggest. s390x would arguably just shove almost everything in -drivers. In reality, I expect most arches to just install both packages anyway.
If you update the ppc64 kernel package, please also do the same for the ppc 32-bit kernel.
There are a few of us out there with vintage PPC hardware. This week, I picked up an IBM 7046-B50 (32-bit only, CHRP), to go with the PPC Macs (G5, G4, G3) that I've been gathering with the intent of recreating a minimal vintage PPC spin (likely based on LXDE). This would include support for other non-Mac 32-bit PPC systems.
While most of the official effort is 64-bit (BE and now LE), the PPC arch still builds 32-bit userspace packages, and has stated that in their wiki that they have no intent to discontinue 32-bit kernels.
Thanks! Al
On Fri, Mar 21, 2014 at 7:54 AM, Al Dunsmuir al.dunsmuir@sympatico.ca wrote:
On Wednesday, March 19, 2014, 2:09:15 PM, Josh Boyer wrote:
On Wed, Mar 19, 2014 at 2:02 PM, Bill Nottingham notting@splat.cc wrote:
Josh Boyer (jwboyer@fedoraproject.org) said:
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops.
Is this even needed on s390 for reasons other than consistency? Similarly with power, is the idea to have a core kernel for running on an LPAR and then -drivers for the rest of it?
Needed? Probably not. At the moment it's not possible to build a normal kernel on one arch and the split on another. If we're going to go off and make changes to anaconda and yum and dnf to cope with this, consistency on what is shipped is probably a good thing.
That being said, it is flexible in terms of the content of those packages. So ppc64 could do what you suggest. s390x would arguably just shove almost everything in -drivers. In reality, I expect most arches to just install both packages anyway.
If you update the ppc64 kernel package, please also do the same for the ppc 32-bit kernel.
I did. I have to adapt for all architectures we build for, and ppc is one of those. You can find it in the scratch build I pointed to earlier in the thread.
josh
On Friday, March 21, 2014, 8:06:41 AM, Josh Boyer wrote:
On Fri, Mar 21, 2014 at 7:54 AM, Al Dunsmuir al.dunsmuir@sympatico.ca wrote:
On Wednesday, March 19, 2014, 2:09:15 PM, Josh Boyer wrote:
On Wed, Mar 19, 2014 at 2:02 PM, Bill Nottingham notting@splat.cc wrote:
Josh Boyer (jwboyer@fedoraproject.org) said:
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
OK, I sorted this out this week. I believe the only arch left to do is s390x and that's only because I forgot about it. Oops.
Is this even needed on s390 for reasons other than consistency? Similarly with power, is the idea to have a core kernel for running on an LPAR and then -drivers for the rest of it?
Needed? Probably not. At the moment it's not possible to build a normal kernel on one arch and the split on another. If we're going to go off and make changes to anaconda and yum and dnf to cope with this, consistency on what is shipped is probably a good thing.
That being said, it is flexible in terms of the content of those packages. So ppc64 could do what you suggest. s390x would arguably just shove almost everything in -drivers. In reality, I expect most arches to just install both packages anyway.
If you update the ppc64 kernel package, please also do the same for the ppc 32-bit kernel.
I did. I have to adapt for all architectures we build for, and ppc is one of those. You can find it in the scratch build I pointed to earlier in the thread.
Thank you kindly for supporting our insanity! I'll grab your scratch build and look closer.
Al
On Mon, Mar 17, 2014 at 03:13:54PM -0400, Josh Boyer wrote:
TODO:
Right now this is what has been building in my COPR for the past week. It works well on x86_64 (which is all I've tested), and i686 builds but I know it's broken on ARM. To finish it up, I need:
- Review from you all.
I took a stab at this.
Let's start with the dumb question first because I couldn't find the obvious answer, why kernel-driver? and not just kernel and kernel-modules-extras? IOW what does kernel-core buy us except for spec churn. I am sure yum is invovled in that answer.
Second, the whole mv .. restore/ and back again dance made me cry. Can we just hack the scripts/Makefile.modinst file to pass a filter argument such that the installs look like
#kernel core make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list #kernel module extras make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list
then you have two directories core/ and suppl/ that you can run depmod on individually and for filelist purposes. Then you can combine it later for packaging purposes I think. Something like that. Heck you might be able to run a depmod check on the filtered files and fail if a dependency isn't copied over during install (or just do it automagically to keep modules.list smaller).
Just a thought.
Most of this is just moving and renaming so that isn't too bad. I didn't focus on the small subtle changes on the %post stuff.
I keep telling myself I would love to rework the spec file to simplify again by moving some of the junk into rpm properly. The number of people who can fully understand the spec file is becoming a dying breed. :-(
So all this churn doesn't make it any better. Hence why I was looking for kernel Makefile solutions or other rpm solutions to simplify the quirks.
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
Much like the config files?
Further testing on all arches.
To make the depmod call fatal to the build. Without making this
fatal we run the risk of pushing kernel-core packages that aren't self-contained and that seems pretty bad to me. It shouldn't be a huge issue as I would think most of the breakage is going to come from merge window stuff and settle down after that.
I was trying to envision a 'make depcheck' or something similar to a 'make configs' that fails early as opposed to an hour after the build completes. But everytime I dive to deep into the Makefile I get lost, try a quick hack and make things worse (I still dream of a 'make kabi' like command for rhel reasons...).
Cheers, Don
josh
diff --git a/kernel.spec b/kernel.spec index e055bb2..4ffa1b6 100644 --- a/kernel.spec +++ b/kernel.spec @@ -34,7 +34,7 @@ Summary: The Linux kernel # For non-released -rc kernels, this will be appended after the rcX and # gitX tags, so a 3 here would become part of release "0.rcX.gitX.3" # -%global baserelease 1 +%global baserelease 8 %global fedora_build %{baserelease}
# base_sublevel is the kernel version we're starting with and patching @@ -385,29 +385,6 @@ Summary: The Linux kernel %define kernel_prereq fileutils, systemd >= 203-2 %define initrd_prereq dracut >= 027
-# -# This macro does requires, provides, conflicts, obsoletes for a kernel package. -# %%kernel_reqprovconf <subpackage> -# It uses any kernel_<subpackage>_conflicts and kernel_<subpackage>_obsoletes -# macros defined above. -# -%define kernel_reqprovconf \ -Provides: kernel = %{rpmversion}-%{pkg_release}\ -Provides: kernel-%{_target_cpu} = %{rpmversion}-%{pkg_release}%{?1:+%{1}}\ -Provides: kernel-drm-nouveau = 16\ -Provides: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\ -Requires(pre): %{kernel_prereq}\ -Requires(pre): %{initrd_prereq}\ -Requires(pre): linux-firmware >= 20130724-29.git31f6b30\ -Requires(preun): systemd >= 200\ -%{expand:%%{?kernel%{?1:_%{1}}_conflicts:Conflicts: %%{kernel%{?1:_%{1}}_conflicts}}}\ -%{expand:%%{?kernel%{?1:_%{1}}_obsoletes:Obsoletes: %%{kernel%{?1:_%{1}}_obsoletes}}}\ -%{expand:%%{?kernel%{?1:_%{1}}_provides:Provides: %%{kernel%{?1:_%{1}}_provides}}}\ -# We can't let RPM do the dependencies automatic because it'll then pick up\ -# a correct but undesirable perl dependency from the module headers which\ -# isn't required for the kernel proper to function\ -AutoReqProv: no\ -%{nil}
Name: kernel%{?variant} Group: System Environment/Kernel @@ -419,8 +396,9 @@ Release: %{pkg_release} # SET %%nobuildarches (ABOVE) INSTEAD ExclusiveArch: noarch %{all_x86} x86_64 ppc ppc64 ppc64p7 s390 s390x %{arm} aarch64 ppc64le ExclusiveOS: Linux +Requires: kernel-%{?variant:%{variant}-}core-uname-r = %{KVERREL}%{?variant} +Requires: kernel-%{?variant:%{variant}-}drivers-uname-r = %{KVERREL}%{?variant}
-%kernel_reqprovconf
# # List the packages used during the kernel build @@ -464,6 +442,7 @@ Source15: merge.pl Source16: mod-extra.list Source17: mod-extra.sh Source18: mod-sign.sh +Source99: filter-modules.sh %define modsign_cmd %{SOURCE18}
Source19: Makefile.release @@ -649,10 +628,31 @@ Patch25044: iwlwifi-dvm-take-mutex-when-sending-SYNC-BT-config-command.patch BuildRoot: %{_tmppath}/kernel-%{KVERREL}-root
%description -The kernel package contains the Linux kernel (vmlinuz), the core of any -Linux operating system. The kernel handles the basic functions -of the operating system: memory allocation, process allocation, device -input and output, etc. +The kernel meta package
+# +# This macro does requires, provides, conflicts, obsoletes for a kernel package. +# %%kernel_reqprovconf <subpackage> +# It uses any kernel_<subpackage>_conflicts and kernel_<subpackage>_obsoletes +# macros defined above. +# +%define kernel_reqprovconf \ +Provides: kernel = %{rpmversion}-%{pkg_release}\ +Provides: kernel-%{_target_cpu} = %{rpmversion}-%{pkg_release}%{?1:+%{1}}\ +Provides: kernel-drm-nouveau = 16\ +Provides: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\ +Requires(pre): %{kernel_prereq}\ +Requires(pre): %{initrd_prereq}\ +Requires(pre): linux-firmware >= 20130724-29.git31f6b30\ +Requires(preun): systemd >= 200\ +%{expand:%%{?kernel%{?1:_%{1}}_conflicts:Conflicts: %%{kernel%{?1:_%{1}}_conflicts}}}\ +%{expand:%%{?kernel%{?1:_%{1}}_obsoletes:Obsoletes: %%{kernel%{?1:_%{1}}_obsoletes}}}\ +%{expand:%%{?kernel%{?1:_%{1}}_provides:Provides: %%{kernel%{?1:_%{1}}_provides}}}\ +# We can't let RPM do the dependencies automatic because it'll then pick up\ +# a correct but undesirable perl dependency from the module headers which\ +# isn't required for the kernel proper to function\ +AutoReqProv: no\ +%{nil}
%package headers Summary: Header files for the Linux kernel for use by glibc @@ -834,53 +834,65 @@ Provides: kernel%{?1:-%{1}}-modules-extra = %{version}-%{release}%{?1:+%{1}}\ Provides: installonlypkg(kernel-module)\ Provides: kernel%{?1:-%{1}}-modules-extra-uname-r = %{KVERREL}%{?1:+%{1}}\ Requires: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\ +Requires: kernel-drivers-uname-r = %{KVERREL}%{?1:+%{1}}\ AutoReqProv: no\ %description -n kernel%{?variant}%{?1:-%{1}}-modules-extra\ This package provides less commonly used kernel modules for the %{?2:%{2} }kernel package.\ %{nil}
# +# This macro creates a kernel-<subpackage>-drivers package. +# %%kernel_drivers_package <subpackage> <pretty-name> +# +%define kernel_drivers_package() \ +%package %{?1:%{1}-}drivers\ +Summary: kernel modules to match the %{?2:%{2}-}core kernel\ +Group: System Environment/Kernel\ +Provides: kernel%{?1:-%{1}}-drivers-%{_target_cpu} = %{version}-%{release}\ +Provides: kernel-drivers-%{_target_cpu} = %{version}-%{release}%{?1:+%{1}}\ +Provides: kernel-drivers = %{version}-%{release}%{?1:+%{1}}\ +Provides: installonlypkg(kernel-module)\ +Provides: kernel-drivers-uname-r = %{KVERREL}%{?1:+%{1}}\ +Requires: kernel-uname-r = %{KVERREL}%{?1:+%{1}}\ +AutoReqProv: no\ +%description -n kernel%{?variant}%{?1:-%{1}}-drivers\ +This package provides commonly used kernel modules for the %{?2:%{2}-}core kernel package.\ +%{nil}
+# # This macro creates a kernel-<subpackage> and its -devel and -debuginfo too. # %%define variant_summary The Linux kernel compiled for <configuration> # %%kernel_variant_package [-n <pretty-name>] <subpackage> # %define kernel_variant_package(n:) \ -%package %1\ +%package %{?1:%{1}-}core\ Summary: %{variant_summary}\ Group: System Environment/Kernel\ -%kernel_reqprovconf\ -%{expand:%%kernel_devel_package %1 %{!?-n:%1}%{?-n:%{-n*}}}\ +Provides: kernel-%{?1:%{1}-}core-uname-r = %{KVERREL}%{?1:+%{1}}\ +%{expand:%%kernel_reqprovconf}\ +%{expand:%%kernel_devel_package %{?1:%{1}} %{!?{-n}:%{1}}%{?{-n}:%{-n*}}}\ +%{expand:%%kernel_drivers_package %{?1:%{1}} %{!?{-n}:%{1}}%{?{-n}:%{-n*}}}\ %if %{with_extra}\ -%{expand:%%kernel_modules_extra_package %1 %{!?-n:%1}%{?-n:%{-n*}}}\ +%{expand:%%kernel_modules_extra_package %{?1:%{1}} %{!?{-n}:%{1}}%{?{-n}:%{-n*}}}\ %endif\ -%{expand:%%kernel_debuginfo_package %1}\ +%{expand:%%kernel_debuginfo_package %{?1:%{1}}}\ %{nil}
-# First the auxiliary packages of the main kernel package. -%kernel_devel_package -%if %{with_extra} -%kernel_modules_extra_package -%endif -%kernel_debuginfo_package
# Now, each variant package.
%define variant_summary The Linux kernel compiled for SMP machines %kernel_variant_package -n SMP smp -%description smp +%description smp-core This package includes a SMP version of the Linux kernel. It is required only on machines with two or more CPUs as well as machines with hyperthreading technology.
Install the kernel-smp package if your machine uses two or more CPUs.
%ifnarch armv7hl %define variant_summary The Linux kernel compiled for PAE capable machines %kernel_variant_package %{pae} -%description %{pae} +%description %{pae}-core This package includes a version of the Linux kernel with support for up to 64GB of high memory. It requires a CPU with Physical Address Extensions (PAE). The non-PAE kernel can only address up to 4GB of memory. @@ -888,7 +900,7 @@ Install the kernel-PAE package if your machine has more than 4GB of memory. %else %define variant_summary The Linux kernel compiled for Cortex-A15 %kernel_variant_package %{pae} -%description %{pae} +%description %{pae}-core This package includes a version of the Linux kernel with support for Cortex-A15 devices with LPAE and HW virtualisation support %endif @@ -897,7 +909,7 @@ Cortex-A15 devices with LPAE and HW virtualisation support %define variant_summary The Linux kernel compiled with extra debugging enabled for PAE capable machines %kernel_variant_package %{pae}debug Obsoletes: kernel-PAE-debug -%description %{pae}debug +%description %{pae}debug-core This package includes a version of the Linux kernel with support for up to 64GB of high memory. It requires a CPU with Physical Address Extensions (PAE). The non-PAE kernel can only address up to 4GB of memory. @@ -910,7 +922,7 @@ on kernel bugs, as some of these options impact performance noticably.
%define variant_summary The Linux kernel compiled with extra debugging enabled %kernel_variant_package debug -%description debug +%description debug-core The kernel package contains the Linux kernel (vmlinuz), the core of any Linux operating system. The kernel handles the basic functions of the operating system: memory allocation, process allocation, device @@ -920,6 +932,16 @@ This variant of the kernel has numerous debugging options enabled. It should only be installed when trying to gather additional information on kernel bugs, as some of these options impact performance noticably.
+# And finally the main -core package
+%define variant_summary The Linux kernel +%kernel_variant_package +%description core +The kernel package contains the Linux kernel (vmlinuz), the core of any +Linux operating system. The kernel handles the basic functions +of the operating system: memory allocation, process allocation, device +input and output, etc.
%prep # do a few sanity-checks for --with *only builds @@ -1589,6 +1611,53 @@ BuildKernel() { %{SOURCE17} $RPM_BUILD_ROOT/lib/modules/$KernelVer %{SOURCE16} %endif
- #
- # Generate the kernel-core and kernel-drivers files lists
- #
- # Copy the System.map file for depmod to use, and create a backup of the
- # full module tree so we can restore it after we're done filtering
- cp System.map $RPM_BUILD_ROOT/.
- pushd $RPM_BUILD_ROOT
- mkdir restore
- cp -r lib/modules/$KernelVer/* restore/.
- # don't include anything going into k-m-e in the file lists
- rm -rf lib/modules/$KernelVer/extra
- # Find all the module files and filter them out into the core and drivers
- # lists. This actually removes anything going into -drivers from the dir.
- find lib/modules/$KernelVer/kernel -name *.ko | sort -n > modules.list
- %{SOURCE99} modules.list
- # Run depmod on the resulting module tree and make sure it isn't broken
- depmod -b . -aeF ./System.map $KernelVer
- # remove files that will be auto generated by depmod at rpm -i time
- pushd $RPM_BUILD_ROOT/lib/modules/$KernelVer/
rm -f modules.{alias*,builtin.bin,dep*,*map,symbols*,devname,softdep}- popd
- # Go back and find all of the various directories in the tree. We use this
- # for the dir lists in kernel-core
- find lib/modules/$KernelVer/kernel -type d | sort -n > module-dirs.list
- # Cleanup
- rm System.map
- cp -r restore/* lib/modules/$KernelVer/.
- rm -rf restore
- popd
- # Make sure the files lists start with absolute paths or rpmbuild fails.
- # Also add in the dir entries
- sed -e 's/^lib*//lib/' $RPM_BUILD_ROOT/k-d.list > ../kernel${Flavour:+-${Flavour}}-drivers.list
- sed -e 's/^lib*/%dir /lib/' $RPM_BUILD_ROOT/module-dirs.list > ../kernel${Flavour:+-${Flavour}}-core.list
- sed -e 's/^lib*//lib/' $RPM_BUILD_ROOT/modules.list >> ../kernel${Flavour:+-${Flavour}}-core.list
- # Cleanup
- rm -f $RPM_BUILD_ROOT/k-d.list
- rm -f $RPM_BUILD_ROOT/modules.list
- rm -f $RPM_BUILD_ROOT/module-dirs.list
%if %{signmodules} # Save the signing keys so we can sign the modules in __modsign_install_post cp signing_key.priv signing_key.priv.sign${Flav} @@ -1857,11 +1926,28 @@ fi\
# # This macro defines a %%post script for a kernel*-modules-extra package. +# It also defines a %%postun script that does the same thing. # %%kernel_modules_extra_post [<subpackage>] # %define kernel_modules_extra_post() \ %{expand:%%post %{?1:%{1}-}modules-extra}\ /sbin/depmod -a %{KVERREL}%{?1:+%{1}}\ +%{nil}\ +%{expand:%%postun %{?1:%{1}-}modules-extra}\ +/sbin/depmod -a %{KVERREL}%{?1:+%{1}}\ +%{nil}
+# +# This macro defines a %%post script for a kernel*-drivers package. +# It also defines a %%postun script that does the same thing. +# %%kernel_drivers_post [<subpackage>] +# +%define kernel_drivers_post() \ +%{expand:%%post %{?1:%{1}-}drivers}\ +/sbin/depmod -a %{KVERREL}%{?1:+%{1}}\ +%{nil}\ +%{expand:%%postun %{?1:%{1}-}drivers}\ +/sbin/depmod -a %{KVERREL}%{?1:+%{1}}\ %{nil}
# This macro defines a %%posttrans script for a kernel package. @@ -1869,7 +1955,7 @@ fi\ # More text can follow to go at the end of this variant's %%post. # %define kernel_variant_posttrans() \ -%{expand:%%posttrans %{?1}}\ +%{expand:%%posttrans %{?1:%{1}-}core}\ /bin/kernel-install add %{KVERREL}%{?1:+%{1}} /%{image_install_path}/vmlinuz-%{KVERREL}%{?1:+%{1}} || exit $?\ %{nil}
@@ -1880,11 +1966,12 @@ fi\ # %define kernel_variant_post(v:r:) \ %{expand:%%kernel_devel_post %{?-v*}}\ +%{expand:%%kernel_drivers_post %{?-v*}}\ %if %{with_extra}\ %{expand:%%kernel_modules_extra_post %{?-v*}}\ %endif\ %{expand:%%kernel_variant_posttrans %{?-v*}}\ -%{expand:%%post %{?-v*}}\ +%{expand:%%post %{?-v*:%{-v*}-}core}\ %{-r:\ if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ] &&\ [ -f /etc/sysconfig/kernel ]; then\ @@ -1897,7 +1984,7 @@ fi}\ # %%kernel_variant_preun <subpackage> # %define kernel_variant_preun() \ -%{expand:%%preun %{?1}}\ +%{expand:%%preun %{?1:%{1}-}core}\ /bin/kernel-install remove %{KVERREL}%{?1:+%{1}} /%{image_install_path}/vmlinuz-%{KVERREL}%{?1:+%{1}} || exit $?\ %{nil}
@@ -1998,6 +2085,10 @@ fi %endif %endif # with_perf
+# empty meta-package +%files +%defattr(-,root,root)
# This is %%{image_install_path} on an arch where that includes ELF files, # or empty otherwise. %define elf_image_install_path %{?kernel_image_elf:%{image_install_path}} @@ -2009,7 +2100,7 @@ fi # %define kernel_variant_files(k:) \ %if %{1}\ -%{expand:%%files %{?2}}\ +%{expand:%%files -f kernel-%{?2:%{2}-}core.list %{?2:%{2}-}core}\ %defattr(-,root,root)\ /%{image_install_path}/%{?-k:%{-k*}}%{!?-k:vmlinuz}-%{KVERREL}%{?2:+%{2}}\ /%{image_install_path}/.vmlinuz-%{KVERREL}%{?2:+%{2}}.hmac \ @@ -2018,9 +2109,10 @@ fi %endif\ %attr(600,root,root) /boot/System.map-%{KVERREL}%{?2:+%{2}}\ /boot/config-%{KVERREL}%{?2:+%{2}}\ +%ghost /boot/initramfs-%{KVERREL}%{?2:+%{2}}.img\ %dir /lib/modules\ %dir /lib/modules/%{KVERREL}%{?2:+%{2}}\ -/lib/modules/%{KVERREL}%{?2:+%{2}}/kernel\ +%dir /lib/modules/%{KVERREL}%{?2:+%{2}}/kernel\ /lib/modules/%{KVERREL}%{?2:+%{2}}/build\ /lib/modules/%{KVERREL}%{?2:+%{2}}/source\ /lib/modules/%{KVERREL}%{?2:+%{2}}/updates\ @@ -2029,7 +2121,8 @@ fi /etc/ld.so.conf.d/kernel-%{KVERREL}%{?2:+%{2}}.conf\ %endif\ /lib/modules/%{KVERREL}%{?2:+%{2}}/modules.*\ -%ghost /boot/initramfs-%{KVERREL}%{?2:+%{2}}.img\ +%{expand:%%files -f kernel-%{?2:%{2}-}drivers.list %{?2:%{2}-}drivers}\ +%defattr(-,root,root)\ %{expand:%%files %{?2:%{2}-}devel}\ %defattr(-,root,root)\ /usr/src/kernels/%{KVERREL}%{?2:+%{2}}\ _______________________________________________ kernel mailing list kernel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/kernel
On Thu, Mar 20, 2014 at 02:13:20PM -0400, Don Zickus wrote:
On Mon, Mar 17, 2014 at 03:13:54PM -0400, Josh Boyer wrote:
TODO:
Right now this is what has been building in my COPR for the past week. It works well on x86_64 (which is all I've tested), and i686 builds but I know it's broken on ARM. To finish it up, I need:
- Review from you all.
I took a stab at this.
Thanks!
Let's start with the dumb question first because I couldn't find the obvious answer, why kernel-driver? and not just kernel and kernel-modules-extras? IOW what does kernel-core buy us except for spec churn. I am sure yum is invovled in that answer.
kernel -> metapackage the requires the equivalent of what kernel is today. kernel-core -> the tiny thing Cloud wants. vmlinux + a small set of drivers kernel-drivers -> the other "half" of what kernel is today kernel-modules-extra -> modules that really aren't needed anywhere by default and I wish we could just turn off entirely.
So kernel-modules-extra already exists today and repurposing it to be the equivalent of kernel-drivers could happen I guess. However, just having "kernel" be the tiny thing is probably broken from a conceptual standpoint because just installing kernel isn't enough to fully boot on most bare-metal machines. If we made it so, then it would probably be large enough to just not bother with this at all. Hence kernel-core.
Second, the whole mv .. restore/ and back again dance made me cry. Can we just hack the scripts/Makefile.modinst file to pass a filter argument such that the installs look like
#kernel core make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list #kernel module extras make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list
then you have two directories core/ and suppl/ that you can run depmod on individually and for filelist purposes. Then you can combine it later for packaging purposes I think. Something like that. Heck you might be able to run a depmod check on the filtered files and fail if a dependency isn't copied over during install (or just do it automagically to keep modules.list smaller).
Er... no? I mean, we could install to core and suppl directories if we had modules.list. But if we had modules.list already then we wouldn't need to generate the %files list, which is what this hunk is doing.
Essentially this is avoiding hardcoded, one line per .ko lists by going through and removing entire directories of things and saving off the list as the contents of kernel-core and kernel-drivers. Then we fix it up since we actually want kernel-drivers to install to /lib/modules/`uname -r`/kernel/ also.
It's probably clearer if you have filter-modules.sh to look at. I've attached it below. Apologies for not including it to begin with.
Most of this is just moving and renaming so that isn't too bad. I didn't focus on the small subtle changes on the %post stuff.
Yeah, that's fairly trivial.
I keep telling myself I would love to rework the spec file to simplify again by moving some of the junk into rpm properly. The number of people who can fully understand the spec file is becoming a dying breed. :-(
So all this churn doesn't make it any better. Hence why I was looking for kernel Makefile solutions or other rpm solutions to simplify the quirks.
Except the complexities don't come from the kernel makefiles. They come from RPM :. It needs a file with a list of files to include for the %files -f directive of both kernel-core and kernel-drivers to work. If we don't use %files -f, we're stuck listing out individual directories and .ko files in the %files section for each. That would be a nightmare to maintain, so I've automated it with %files -f.
Some of this could be avoided (not a whole lot) if we invented /lib/modules/`uname -r`/suppl and installed the kernel-drivers content there, but that's yet-another location for things. I'm not overly thrilled with doing that. All it would really avoid is the restore we do.
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
Much like the config files?
Eh, not really. It's a list of directories and individual .ko files to filter out. Example filter-x86_64.sh file also below.
Further testing on all arches.
To make the depmod call fatal to the build. Without making this
fatal we run the risk of pushing kernel-core packages that aren't self-contained and that seems pretty bad to me. It shouldn't be a huge issue as I would think most of the breakage is going to come from merge window stuff and settle down after that.
I was trying to envision a 'make depcheck' or something similar to a 'make configs' that fails early as opposed to an hour after the build completes. But everytime I dive to deep into the Makefile I get lost, try a quick hack and make things worse (I still dream of a 'make kabi' like command for rhel reasons...).
That... wouldn't help here. The configs are untouched, and Kconfig should already gripe if you have a config set that doesn't have a dep set. You actually need the built modules in the locations they're going to be installed at in order to run depmod and ensure the required .ko _files_ are present. I really don't see a way to avoid building the drivers here.
josh
diff --git a/filter-modules.sh b/filter-modules.sh new file mode 100755 index 0000000..b06a8b9 --- /dev/null +++ b/filter-modules.sh @@ -0,0 +1,111 @@ +#! /bin/bash +# +# Called as filter-modules.sh list-of-modules Arch + +# Grab the arch-specific filter lists +source ./filter-$2.sh + +filter_dir() { + filelist=$1 + dir=$2 + + grep -v -e "${dir}/" ${filelist} > ${filelist}.tmp + + if [ $? -ne 0 ] + then + echo "Couldn't remove ${dir}. Skipping." + else + grep -e "${dir}/" ${filelist} >> k-d.list + mv ${filelist}.tmp $filelist + fi + + return 0 +} + +filter_ko() { + filelist=$1 + mod=$2 + + grep -v -e "${mod}.ko" ${filelist} > ${filelist}.tmp + + if [ $? -ne 0 ] + then + echo "Couldn't remove ${mod}.ko Skipping." + else + grep -e "${mod}.ko" ${filelist} >> k-d.list + mv ${filelist}.tmp $filelist + fi + + return 0 +} + +# Filter the drivers/ subsystems +for subsys in ${driverdirs} +do + filter_dir $1 drivers/${subsys} +done + +# Filter the networking drivers +for netdrv in ${netdrvs} +do + filter_dir $1 drivers/net/${netdrv} +done + +# Filter the ethernet drivers +for eth in ${ethdrvs} +do + filter_dir $1 drivers/net/ethernet/${eth} +done + +# SCSI +for scsi in ${scsidrvs} +do + filter_dir $1 drivers/scsi/${scsi} +done + +# TTY +for tty in ${ttydrvs} +do + filter_dir $1 drivers/tty/${tty} +done + +# USB +for usb in ${usbdrvs} +do + filter_dir $1 drivers/usb/${usb} +done + +# Filesystems +for fs in ${fsdrvs} +do + filter_dir $1 fs/${fs} +done + +# Network protocols +for prot in ${netprots} +do + filter_dir $1 kernel/net/${prot} +done + +# DRM +for drm in ${drmdrvs} +do + filter_dir $1 drivers/gpu/drm/${drm} +done + +# Just kill sound. +filter_dir $1 kernel/sound + +# Now go through and filter any single .ko files that might have deps on the +# things we filtered above +for mod in ${singlemods} +do + filter_ko $1 ${mod} +done + +# Go through our generated drivers list and remove the .ko files. We'll +# restore them later. +for mod in `cat k-d.list` +do + rm -rf $mod +done diff --git a/filter-x86_64.sh b/filter-x86_64.sh new file mode 100644 index 0000000..4d9fe44 --- /dev/null +++ b/filter-x86_64.sh @@ -0,0 +1,21 @@ +#! /bin/bash + +driverdirs="atm auxdisplay bcma bluetooth fmc iio infiniband isdn leds media memstick message mfd mmc mtd nfc ntb pcmcia platform power ssb staging uio uwb" + +netdrvs="appletalk dsa hamradio ieee802154 irda ppp slip usb wireless" + +ethdrvs="3com adaptec alteon amd atheros broadcom cadence calxeda chelsio cisco dec dlink emulex icplus marvell mellanox neterion nvidia oki-semi packetengines qlogic rdc renesas sfc silan sis smsc stmicro sun tehuti ti wiznet xircom" + +scsidrvs="aacraid aic7xxx aic94xx be2iscsi bfa bnx2i bnx2fc csiostor cxgbi esas2r fcoe fnic isci libsas lpfc megaraid mpt2sas mpt3sas mvsas pm8001 qla2xxx qla4xxx sym53c8xx_2 ufs" + +ttydrvs="ipwireless" + +usbdrvs="atm wusbcore" + +fsdrvs="affs befs coda cramfs dlm ecryptfs hfs hfsplus isofs jfs minix ncpfs nilfs2 ocfs2 reiserfs romfs squashfs sysv ubifs udf ufs" + +netprots="appletalk atm ax25 batman-adv bluetooth dccp dsa ieee802154 irda l2tp mac80211 mac802154 netrom nfc rds rfkill rose sctp wireless" + +drmdrvs="ast gma500 mgag200 via nouveau" + +singlemods="ntb_netdev iscsi_ibft iscsi_boot_sysfs iscsi_tcp megaraid pmcraid qla1280 9pnet_rdma svcrdma xprtrdma hid-picolcd hid-prodikeys hwa-hc hwpoison-inject hid-sensor-hub"
On Thu, Mar 20, 2014 at 02:51:13PM -0400, Josh Boyer wrote:
On Thu, Mar 20, 2014 at 02:13:20PM -0400, Don Zickus wrote:
On Mon, Mar 17, 2014 at 03:13:54PM -0400, Josh Boyer wrote:
TODO:
Right now this is what has been building in my COPR for the past week. It works well on x86_64 (which is all I've tested), and i686 builds but I know it's broken on ARM. To finish it up, I need:
- Review from you all.
I took a stab at this.
Thanks!
Let's start with the dumb question first because I couldn't find the obvious answer, why kernel-driver? and not just kernel and kernel-modules-extras? IOW what does kernel-core buy us except for spec churn. I am sure yum is invovled in that answer.
kernel -> metapackage the requires the equivalent of what kernel is today. kernel-core -> the tiny thing Cloud wants. vmlinux + a small set of drivers kernel-drivers -> the other "half" of what kernel is today kernel-modules-extra -> modules that really aren't needed anywhere by default and I wish we could just turn off entirely.
So kernel-modules-extra already exists today and repurposing it to be the equivalent of kernel-drivers could happen I guess. However, just having "kernel" be the tiny thing is probably broken from a conceptual standpoint because just installing kernel isn't enough to fully boot on most bare-metal machines. If we made it so, then it would probably be large enough to just not bother with this at all. Hence kernel-core.
Right, I guess I don't understand how kernel-drivers magically installs/not installs. I had assumed some of this was hardcoded in some install file anaconda reads. So my initial assumption was kernel and kernel-drivers would auto install on a normal desktop. I didn't think far enough to understand how an update would work..
So in this case, I suppose cloud only installs kernel-core whereas desktop installs kernel (which installs both -core and -driver)?
Second, the whole mv .. restore/ and back again dance made me cry. Can we just hack the scripts/Makefile.modinst file to pass a filter argument such that the installs look like
#kernel core make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list #kernel module extras make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list
then you have two directories core/ and suppl/ that you can run depmod on individually and for filelist purposes. Then you can combine it later for packaging purposes I think. Something like that. Heck you might be able to run a depmod check on the filtered files and fail if a dependency isn't copied over during install (or just do it automagically to keep modules.list smaller).
Er... no? I mean, we could install to core and suppl directories if we had modules.list. But if we had modules.list already then we wouldn't need to generate the %files list, which is what this hunk is doing.
Essentially this is avoiding hardcoded, one line per .ko lists by going through and removing entire directories of things and saving off the list as the contents of kernel-core and kernel-drivers. Then we fix it up since we actually want kernel-drivers to install to /lib/modules/`uname -r`/kernel/ also.
It's probably clearer if you have filter-modules.sh to look at. I've attached it below. Apologies for not including it to begin with.
Ah, yes, now things are clearer. I had assumed you were listing everything one by one (sorta akin to what is done with the config options).
I could argue you would have better control if you mimic'd the config options and created a modules.list. But I am not maintaining this, so I won't. :-)
Most of this is just moving and renaming so that isn't too bad. I didn't focus on the small subtle changes on the %post stuff.
Yeah, that's fairly trivial.
I keep telling myself I would love to rework the spec file to simplify again by moving some of the junk into rpm properly. The number of people who can fully understand the spec file is becoming a dying breed. :-(
So all this churn doesn't make it any better. Hence why I was looking for kernel Makefile solutions or other rpm solutions to simplify the quirks.
Except the complexities don't come from the kernel makefiles. They come from RPM :. It needs a file with a list of files to include for the %files -f directive of both kernel-core and kernel-drivers to work. If we don't use %files -f, we're stuck listing out individual directories and .ko files in the %files section for each. That would be a nightmare to maintain, so I've automated it with %files -f.
I wasn't referring to the %files stuff. More like all the shell scripting prep work for restore/ and friends. Just trying to think of ways to simplify things.
Some of this could be avoided (not a whole lot) if we invented /lib/modules/`uname -r`/suppl and installed the kernel-drivers content there, but that's yet-another location for things. I'm not overly thrilled with doing that. All it would really avoid is the restore we do.
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
Much like the config files?
Eh, not really. It's a list of directories and individual .ko files to filter out. Example filter-x86_64.sh file also below.
Further testing on all arches.
To make the depmod call fatal to the build. Without making this
fatal we run the risk of pushing kernel-core packages that aren't self-contained and that seems pretty bad to me. It shouldn't be a huge issue as I would think most of the breakage is going to come from merge window stuff and settle down after that.
I was trying to envision a 'make depcheck' or something similar to a 'make configs' that fails early as opposed to an hour after the build completes. But everytime I dive to deep into the Makefile I get lost, try a quick hack and make things worse (I still dream of a 'make kabi' like command for rhel reasons...).
That... wouldn't help here. The configs are untouched, and Kconfig should already gripe if you have a config set that doesn't have a dep set. You actually need the built modules in the locations they're going to be installed at in order to run depmod and ensure the required .ko _files_ are present. I really don't see a way to avoid building the drivers here.
I didn't explain myself clearly. The idea was if you had a list, you could somehow run it through Kconfig (or Makefile) and check that all the deps were on the list and fail at pre-build time with a broken dependency on your module.list. However, with the way you are doing it, this obviously can't work as you don't know what is on your list until after you build.
Thanks for the filter-modules.sh script, it cleared things up.
Cheers, Don
josh
diff --git a/filter-modules.sh b/filter-modules.sh new file mode 100755 index 0000000..b06a8b9 --- /dev/null +++ b/filter-modules.sh @@ -0,0 +1,111 @@ +#! /bin/bash +# +# Called as filter-modules.sh list-of-modules Arch
+# Grab the arch-specific filter lists +source ./filter-$2.sh
+filter_dir() {
- filelist=$1
- dir=$2
- grep -v -e "${dir}/" ${filelist} > ${filelist}.tmp
- if [ $? -ne 0 ]
- then
echo "Couldn't remove ${dir}. Skipping."- else
grep -e "${dir}/" ${filelist} >> k-d.listmv ${filelist}.tmp $filelist- fi
- return 0
+}
+filter_ko() {
- filelist=$1
- mod=$2
- grep -v -e "${mod}.ko" ${filelist} > ${filelist}.tmp
- if [ $? -ne 0 ]
- then
echo "Couldn't remove ${mod}.ko Skipping."- else
grep -e "${mod}.ko" ${filelist} >> k-d.listmv ${filelist}.tmp $filelist- fi
- return 0
+}
+# Filter the drivers/ subsystems +for subsys in ${driverdirs} +do
- filter_dir $1 drivers/${subsys}
+done
+# Filter the networking drivers +for netdrv in ${netdrvs} +do
- filter_dir $1 drivers/net/${netdrv}
+done
+# Filter the ethernet drivers +for eth in ${ethdrvs} +do
- filter_dir $1 drivers/net/ethernet/${eth}
+done
+# SCSI +for scsi in ${scsidrvs} +do
- filter_dir $1 drivers/scsi/${scsi}
+done
+# TTY +for tty in ${ttydrvs} +do
- filter_dir $1 drivers/tty/${tty}
+done
+# USB +for usb in ${usbdrvs} +do
- filter_dir $1 drivers/usb/${usb}
+done
+# Filesystems +for fs in ${fsdrvs} +do
- filter_dir $1 fs/${fs}
+done
+# Network protocols +for prot in ${netprots} +do
- filter_dir $1 kernel/net/${prot}
+done
+# DRM +for drm in ${drmdrvs} +do
- filter_dir $1 drivers/gpu/drm/${drm}
+done
+# Just kill sound. +filter_dir $1 kernel/sound
+# Now go through and filter any single .ko files that might have deps on the +# things we filtered above +for mod in ${singlemods} +do
filter_ko $1 ${mod}+done
+# Go through our generated drivers list and remove the .ko files. We'll +# restore them later. +for mod in `cat k-d.list` +do
- rm -rf $mod
+done diff --git a/filter-x86_64.sh b/filter-x86_64.sh new file mode 100644 index 0000000..4d9fe44 --- /dev/null +++ b/filter-x86_64.sh @@ -0,0 +1,21 @@ +#! /bin/bash
+driverdirs="atm auxdisplay bcma bluetooth fmc iio infiniband isdn leds media memstick message mfd mmc mtd nfc ntb pcmcia platform power ssb staging uio uwb"
+netdrvs="appletalk dsa hamradio ieee802154 irda ppp slip usb wireless"
+ethdrvs="3com adaptec alteon amd atheros broadcom cadence calxeda chelsio cisco dec dlink emulex icplus marvell mellanox neterion nvidia oki-semi packetengines qlogic rdc renesas sfc silan sis smsc stmicro sun tehuti ti wiznet xircom"
+scsidrvs="aacraid aic7xxx aic94xx be2iscsi bfa bnx2i bnx2fc csiostor cxgbi esas2r fcoe fnic isci libsas lpfc megaraid mpt2sas mpt3sas mvsas pm8001 qla2xxx qla4xxx sym53c8xx_2 ufs"
+ttydrvs="ipwireless"
+usbdrvs="atm wusbcore"
+fsdrvs="affs befs coda cramfs dlm ecryptfs hfs hfsplus isofs jfs minix ncpfs nilfs2 ocfs2 reiserfs romfs squashfs sysv ubifs udf ufs"
+netprots="appletalk atm ax25 batman-adv bluetooth dccp dsa ieee802154 irda l2tp mac80211 mac802154 netrom nfc rds rfkill rose sctp wireless"
+drmdrvs="ast gma500 mgag200 via nouveau"
+singlemods="ntb_netdev iscsi_ibft iscsi_boot_sysfs iscsi_tcp megaraid pmcraid qla1280 9pnet_rdma svcrdma xprtrdma hid-picolcd hid-prodikeys hwa-hc hwpoison-inject hid-sensor-hub"
On Thu, Mar 20, 2014 at 03:29:12PM -0400, Don Zickus wrote:
On Thu, Mar 20, 2014 at 02:51:13PM -0400, Josh Boyer wrote:
Let's start with the dumb question first because I couldn't find the obvious answer, why kernel-driver? and not just kernel and kernel-modules-extras? IOW what does kernel-core buy us except for spec churn. I am sure yum is invovled in that answer.
kernel -> metapackage the requires the equivalent of what kernel is today. kernel-core -> the tiny thing Cloud wants. vmlinux + a small set of drivers kernel-drivers -> the other "half" of what kernel is today kernel-modules-extra -> modules that really aren't needed anywhere by default and I wish we could just turn off entirely.
So kernel-modules-extra already exists today and repurposing it to be the equivalent of kernel-drivers could happen I guess. However, just having "kernel" be the tiny thing is probably broken from a conceptual standpoint because just installing kernel isn't enough to fully boot on most bare-metal machines. If we made it so, then it would probably be large enough to just not bother with this at all. Hence kernel-core.
Right, I guess I don't understand how kernel-drivers magically installs/not installs. I had assumed some of this was hardcoded in some install file anaconda reads. So my initial assumption was kernel and kernel-drivers would auto install on a normal desktop. I didn't think far enough to understand how an update would work..
I believe anaconda does install "kernel" today. So since the kernel metapackage Requires:kernel-drivers, it will bring that in automatically. Updates from existing installs would pick up kernel, kernel-core, and kernel-drivers based on the Requires: present.
It looks like 3 packages instead of one, but the overall installed size is literally the same as the existing kernel package today. It's a split to keep things working, while allowing new approaches at the same time.
So in this case, I suppose cloud only installs kernel-core whereas desktop installs kernel (which installs both -core and -driver)?
Yes, exactly. That would be done via kickstart (and is extendable to anyone that wants to avoid kernel-drivers but Cloud is the main user.)
Second, the whole mv .. restore/ and back again dance made me cry. Can we just hack the scripts/Makefile.modinst file to pass a filter argument such that the installs look like
#kernel core make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list #kernel module extras make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list
then you have two directories core/ and suppl/ that you can run depmod on individually and for filelist purposes. Then you can combine it later for packaging purposes I think. Something like that. Heck you might be able to run a depmod check on the filtered files and fail if a dependency isn't copied over during install (or just do it automagically to keep modules.list smaller).
Er... no? I mean, we could install to core and suppl directories if we had modules.list. But if we had modules.list already then we wouldn't need to generate the %files list, which is what this hunk is doing.
Essentially this is avoiding hardcoded, one line per .ko lists by going through and removing entire directories of things and saving off the list as the contents of kernel-core and kernel-drivers. Then we fix it up since we actually want kernel-drivers to install to /lib/modules/`uname -r`/kernel/ also.
It's probably clearer if you have filter-modules.sh to look at. I've attached it below. Apologies for not including it to begin with.
Ah, yes, now things are clearer. I had assumed you were listing everything one by one (sorta akin to what is done with the config options).
I could argue you would have better control if you mimic'd the config options and created a modules.list. But I am not maintaining this, so I won't. :-)
I might be able to generate a list of modules (.ko files) based on the config settings. That would basically be the equivalent of the first find we do in the spec file change. Not sure it would simplify anything else, but I'll think about it a bit and see if there's further optimizations to be done.
Most of this is just moving and renaming so that isn't too bad. I didn't focus on the small subtle changes on the %post stuff.
Yeah, that's fairly trivial.
I keep telling myself I would love to rework the spec file to simplify again by moving some of the junk into rpm properly. The number of people who can fully understand the spec file is becoming a dying breed. :-(
So all this churn doesn't make it any better. Hence why I was looking for kernel Makefile solutions or other rpm solutions to simplify the quirks.
Except the complexities don't come from the kernel makefiles. They come from RPM :. It needs a file with a list of files to include for the %files -f directive of both kernel-core and kernel-drivers to work. If we don't use %files -f, we're stuck listing out individual directories and .ko files in the %files section for each. That would be a nightmare to maintain, so I've automated it with %files -f.
I wasn't referring to the %files stuff. More like all the shell scripting prep work for restore/ and friends. Just trying to think of ways to simplify things.
Yeah, I understand. I tried to keep it as simple as possible and that's what I wound up with. Should have seen it before I realized it was easier to just restore things.
Could move it into a script we call instead of listing it out in the spec file ;).
Some of this could be avoided (not a whole lot) if we invented /lib/modules/`uname -r`/suppl and installed the kernel-drivers content there, but that's yet-another location for things. I'm not overly thrilled with doing that. All it would really avoid is the restore we do.
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
Much like the config files?
Eh, not really. It's a list of directories and individual .ko files to filter out. Example filter-x86_64.sh file also below.
Further testing on all arches.
To make the depmod call fatal to the build. Without making this
fatal we run the risk of pushing kernel-core packages that aren't self-contained and that seems pretty bad to me. It shouldn't be a huge issue as I would think most of the breakage is going to come from merge window stuff and settle down after that.
I was trying to envision a 'make depcheck' or something similar to a 'make configs' that fails early as opposed to an hour after the build completes. But everytime I dive to deep into the Makefile I get lost, try a quick hack and make things worse (I still dream of a 'make kabi' like command for rhel reasons...).
That... wouldn't help here. The configs are untouched, and Kconfig should already gripe if you have a config set that doesn't have a dep set. You actually need the built modules in the locations they're going to be installed at in order to run depmod and ensure the required .ko _files_ are present. I really don't see a way to avoid building the drivers here.
I didn't explain myself clearly. The idea was if you had a list, you could somehow run it through Kconfig (or Makefile) and check that all the deps were on the list and fail at pre-build time with a broken dependency on your module.list. However, with the way you are doing it, this obviously can't work as you don't know what is on your list until after you build.
Even ignoring what I'm doing here, I don't see how that works. You can have a pre-made list of drivers, but the depmod check depends on what is installed and when. If you build both foo.ko and bar.ko, but you only install foo.ko and not it's dep bar.ko, then you're broken. Doing it before you build and install the drivers doesn't seem feasible.
It is quite possible we are talking about entirely different kinds of checks and I'm just not realizing it.
josh
On Thu, Mar 20, 2014 at 03:47:19PM -0400, Josh Boyer wrote:
I believe anaconda does install "kernel" today. So since the kernel metapackage Requires:kernel-drivers, it will bring that in automatically. Updates from existing installs would pick up kernel, kernel-core, and kernel-drivers based on the Requires: present.
It looks like 3 packages instead of one, but the overall installed size is literally the same as the existing kernel package today. It's a split to keep things working, while allowing new approaches at the same time.
So in this case, I suppose cloud only installs kernel-core whereas desktop installs kernel (which installs both -core and -driver)?
Yes, exactly. That would be done via kickstart (and is extendable to anyone that wants to avoid kernel-drivers but Cloud is the main user.)
Thanks!
Second, the whole mv .. restore/ and back again dance made me cry. Can we just hack the scripts/Makefile.modinst file to pass a filter argument such that the installs look like
#kernel core make modules_install INSTALL_MOD_DIR=core INSTALL_MOD_FILTER=modules.list #kernel module extras make modules_install INSTALL_MOD_DIR=suppl INSTALL_MOD_FILTER_INV=modules.list
then you have two directories core/ and suppl/ that you can run depmod on individually and for filelist purposes. Then you can combine it later for packaging purposes I think. Something like that. Heck you might be able to run a depmod check on the filtered files and fail if a dependency isn't copied over during install (or just do it automagically to keep modules.list smaller).
Er... no? I mean, we could install to core and suppl directories if we had modules.list. But if we had modules.list already then we wouldn't need to generate the %files list, which is what this hunk is doing.
Essentially this is avoiding hardcoded, one line per .ko lists by going through and removing entire directories of things and saving off the list as the contents of kernel-core and kernel-drivers. Then we fix it up since we actually want kernel-drivers to install to /lib/modules/`uname -r`/kernel/ also.
It's probably clearer if you have filter-modules.sh to look at. I've attached it below. Apologies for not including it to begin with.
Ah, yes, now things are clearer. I had assumed you were listing everything one by one (sorta akin to what is done with the config options).
I could argue you would have better control if you mimic'd the config options and created a modules.list. But I am not maintaining this, so I won't. :-)
I might be able to generate a list of modules (.ko files) based on the config settings. That would basically be the equivalent of the first find we do in the spec file change. Not sure it would simplify anything else, but I'll think about it a bit and see if there's further optimizations to be done.
It was just a different point of view. Not saying it is any better. Just thought it might simplify the spec file changes and control the kernel-core bloat over a number of releases.
The thought was if you the average desktop user was install kenrel-core and kernel-driver, then it doesn't matter where the .kos are because in the end, all the .kos end up on the machine. So the kernel-core module.list would really be for cloud. And someone could keep trimming that list until the cloud folks had something super small that made them excited. If the list was too small it didn't impact the desktop folks (not that your approach causes any negative impact either).
Again it was just solving a problem from a different perspective. If you are happy with your approach continue with it. I'll have to see how it works in RHEL in a couple of years. :-)
Most of this is just moving and renaming so that isn't too bad. I didn't focus on the small subtle changes on the %post stuff.
Yeah, that's fairly trivial.
I keep telling myself I would love to rework the spec file to simplify again by moving some of the junk into rpm properly. The number of people who can fully understand the spec file is becoming a dying breed. :-(
So all this churn doesn't make it any better. Hence why I was looking for kernel Makefile solutions or other rpm solutions to simplify the quirks.
Except the complexities don't come from the kernel makefiles. They come from RPM :. It needs a file with a list of files to include for the %files -f directive of both kernel-core and kernel-drivers to work. If we don't use %files -f, we're stuck listing out individual directories and .ko files in the %files section for each. That would be a nightmare to maintain, so I've automated it with %files -f.
I wasn't referring to the %files stuff. More like all the shell scripting prep work for restore/ and friends. Just trying to think of ways to simplify things.
Yeah, I understand. I tried to keep it as simple as possible and that's what I wound up with. Should have seen it before I realized it was easier to just restore things.
Could move it into a script we call instead of listing it out in the spec file ;).
Ugh. Like rpmmacros and its brethren? I prefer seeing the mess. :-)
Some of this could be avoided (not a whole lot) if we invented /lib/modules/`uname -r`/suppl and installed the kernel-drivers content there, but that's yet-another location for things. I'm not overly thrilled with doing that. All it would really avoid is the restore we do.
- A per-arch filter list, because the existing one that works on
x86_64 leaves modules in kernel-core on ARM that lack their dependencies. Bad.
Much like the config files?
Eh, not really. It's a list of directories and individual .ko files to filter out. Example filter-x86_64.sh file also below.
Further testing on all arches.
To make the depmod call fatal to the build. Without making this
fatal we run the risk of pushing kernel-core packages that aren't self-contained and that seems pretty bad to me. It shouldn't be a huge issue as I would think most of the breakage is going to come from merge window stuff and settle down after that.
I was trying to envision a 'make depcheck' or something similar to a 'make configs' that fails early as opposed to an hour after the build completes. But everytime I dive to deep into the Makefile I get lost, try a quick hack and make things worse (I still dream of a 'make kabi' like command for rhel reasons...).
That... wouldn't help here. The configs are untouched, and Kconfig should already gripe if you have a config set that doesn't have a dep set. You actually need the built modules in the locations they're going to be installed at in order to run depmod and ensure the required .ko _files_ are present. I really don't see a way to avoid building the drivers here.
I didn't explain myself clearly. The idea was if you had a list, you could somehow run it through Kconfig (or Makefile) and check that all the deps were on the list and fail at pre-build time with a broken dependency on your module.list. However, with the way you are doing it, this obviously can't work as you don't know what is on your list until after you build.
Even ignoring what I'm doing here, I don't see how that works. You can have a pre-made list of drivers, but the depmod check depends on what is installed and when. If you build both foo.ko and bar.ko, but you only install foo.ko and not it's dep bar.ko, then you're broken. Doing it before you build and install the drivers doesn't seem feasible.
It is quite possible we are talking about entirely different kinds of checks and I'm just not realizing it.
The thought had nothing to do with the 'depmod' binary. It was more to do with the idea that Kconfig knew some deps and perhaps that could be leveraged in a pre-build way. But in the end, it is really the EXPORT_SYMBOLS that create the deps and those are gathered in the .ko post-build (which is what depmod utilizes anyway). So my idea might not have been successful. Some things sound interesting until you type them out and realize they don't work.
Cheers, Don
kernel@lists.fedoraproject.org
-
Al Dunsmuir -
Bill Nottingham -
Don Zickus -
Josh Boyer