Hi.
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
There is a /etc/sysconfig/kernel, how about dropping a "HARDLINK=yes/no" there?
On Tue, 31 Jan 2006, Ralf Ertzinger wrote:
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
There is a /etc/sysconfig/kernel, how about dropping a "HARDLINK=yes/no" there?
that would be great. So far I only ended up with foobarred kernel trees after removing/adding lots of kernel rpms.
Paul
On Tue, Jan 31, 2006 at 07:44:17PM +0100, Ralf Ertzinger wrote:
Hi.
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
only if you have a huge number of kernels installed, which shouldn't be the case.
There is a /etc/sysconfig/kernel, how about dropping a "HARDLINK=yes/no" there?
It would mean adding parsing to the kernel spec file, which I'm not entirely enthusiastic about, especially for something that isn't the common case.
With installonlyn being enabled by default in rawhide yum, there should never be >3 kernel-devel's installed.
Dave
On Tue, 31 Jan 2006, Dave Jones wrote:
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
only if you have a huge number of kernels installed, which shouldn't be the case.
The only gain for hardlinks *is* when you have a lot of kernels installed. Also, I don't think yum uninstalls any kernels, so a machine which is updated reguarly can end up with a lot of kernels. Especially if smp and up kernels are present.
Also, "normal" systems wouldn't need the kernel source. the stuff in /lib/modules should be enough to compile modules.
But my problem i guess is mostly in that I've had a few incompletele kernel trees and couldn't compile from them anymore.
The process is terribly slow on older hardware though.
I would happy if the kernel rpms checked for the hardlinks rpm and skipped it when the rpm isnt installed.
Paul
On Wed, Feb 01, 2006 at 12:13:38AM +0100, Paul Wouters wrote:
On Tue, 31 Jan 2006, Dave Jones wrote:
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
only if you have a huge number of kernels installed, which shouldn't be the case.
The only gain for hardlinks *is* when you have a lot of kernels installed.
it's a space gain as soon as you've >1 installed.
Also, I don't think yum uninstalls any kernels
it does with rawhide kernels. seen installonlyn plugin.
, so a machine which is updated reguarly can end up with a lot of kernels. Especially if smp and up kernels are present.
rawhide only installs the one thats needed.
But my problem i guess is mostly in that I've had a few incompletele kernel trees and couldn't compile from them anymore.
Are you trying to compile a kernel from a kernel-devel tree ? That won't work, nor should it. That's why there's a .src.rpm
The process is terribly slow on older hardware though.
I would happy if the kernel rpms checked for the hardlinks rpm and skipped it when the rpm isnt installed.
It does. However we have a Prereq: /usr/sbin/hardlink in the spec too. I'll nuke that, and it'll then allow you to rpm -e hardlink, and things should make everyone happy.
Dave
On Tue, 2006-01-31 at 17:21 -0500, Dave Jones wrote:
On Tue, Jan 31, 2006 at 07:44:17PM +0100, Ralf Ertzinger wrote:
Hi.
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
only if you have a huge number of kernels installed, which shouldn't be the case.
Or if you build kernel modules against multiple kernel VR's.
There is a /etc/sysconfig/kernel, how about dropping a "HARDLINK=yes/no" there?
It would mean adding parsing to the kernel spec file, which I'm not entirely enthusiastic about, especially for something that isn't the common case.
%post [ -f /etc/sysconfig/kernel ] && . /etc/sysconfig/kernel if [ "$HARDLINK" != "no" -a -x /usr/sbin/hardlink ] ; then pushd /usr/src/kernels/2.6.15-1.1823_FC4-i686 > /dev/null /usr/bin/find . -type f | while read f; do hardlink -c /usr/src/kernels/*FC*/$f $f ; done popd > /dev/null fi
On Tue, Jan 31, 2006 at 07:05:53PM -0500, Ignacio Vazquez-Abrams wrote:
On Tue, 2006-01-31 at 17:21 -0500, Dave Jones wrote:
On Tue, Jan 31, 2006 at 07:44:17PM +0100, Ralf Ertzinger wrote:
Hi.
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
only if you have a huge number of kernels installed, which shouldn't be the case.
Or if you build kernel modules against multiple kernel VR's.
There is a /etc/sysconfig/kernel, how about dropping a "HARDLINK=yes/no" there?
It would mean adding parsing to the kernel spec file, which I'm not entirely enthusiastic about, especially for something that isn't the common case.
%post [ -f /etc/sysconfig/kernel ] && . /etc/sysconfig/kernel if [ "$HARDLINK" != "no" -a -x /usr/sbin/hardlink ] ; then pushd /usr/src/kernels/2.6.15-1.1823_FC4-i686 > /dev/null /usr/bin/find . -type f | while read f; do hardlink -c /usr/src/kernels/*FC*/$f $f ; done popd > /dev/null fi
Removing the Prereq: /usr/sbin/hardlink from the spec (which I just did in CVS) should have the effect that you can now rpm -e hardlink, and it'll do what you want, whilst leaving it enabled for users by default.
Dave
On Tue, 2006-01-31 at 19:25 -0500, Dave Jones wrote:
Removing the Prereq: /usr/sbin/hardlink from the spec (which I just did in CVS) should have the effect that you can now rpm -e hardlink, and it'll do what you want, whilst leaving it enabled for users by default.
And making it so that either 1) you don't have hardlink, or 2) forcing you to uninstall it, install kernel-devel, and then reinstall it.
On Tue, 2006-01-31 at 19:25 -0500, Dave Jones wrote:
Removing the Prereq: /usr/sbin/hardlink from the spec (which I just did in CVS) should have the effect that you can now rpm -e hardlink, and it'll do what you want, whilst leaving it enabled for users by default.
Yeah, but that sucks, because I want hardlink for things that *really* take lots of space.
(like, you know, mirroring rawhide for 2 or 3 days)
Peter Jones wrote :
On Tue, 2006-01-31 at 19:25 -0500, Dave Jones wrote:
Removing the Prereq: /usr/sbin/hardlink from the spec (which I just did in CVS) should have the effect that you can now rpm -e hardlink, and it'll do what you want, whilst leaving it enabled for users by default.
Yeah, but that sucks, because I want hardlink for things that *really* take lots of space.
(like, you know, mirroring rawhide for 2 or 3 days)
Install hardlink++ ... it's even faster! :-)
Matthias
On Wed, 2006-02-01 at 16:53 +0100, Matthias Saou wrote:
Peter Jones wrote :
On Tue, 2006-01-31 at 19:25 -0500, Dave Jones wrote:
Removing the Prereq: /usr/sbin/hardlink from the spec (which I just did in CVS) should have the effect that you can now rpm -e hardlink, and it'll do what you want, whilst leaving it enabled for users by default.
Yeah, but that sucks, because I want hardlink for things that *really* take lots of space.
(like, you know, mirroring rawhide for 2 or 3 days)
Install hardlink++ ... it's even faster! :-)
You're not wrong, but this suggestion is ignoring the point.
The point was that having the kernel's %post decide you do or don't want something based on if an unrelated package is installed isn't a good plan at all.
On Wed, Feb 01, 2006 at 12:23:42PM -0500, Peter Jones wrote:
On Wed, 2006-02-01 at 16:53 +0100, Matthias Saou wrote:
Peter Jones wrote :
On Tue, 2006-01-31 at 19:25 -0500, Dave Jones wrote:
Removing the Prereq: /usr/sbin/hardlink from the spec (which I just did in CVS) should have the effect that you can now rpm -e hardlink, and it'll do what you want, whilst leaving it enabled for users by default.
Yeah, but that sucks, because I want hardlink for things that *really* take lots of space.
(like, you know, mirroring rawhide for 2 or 3 days)
Install hardlink++ ... it's even faster! :-)
You're not wrong, but this suggestion is ignoring the point.
The point was that having the kernel's %post decide you do or don't want something based on if an unrelated package is installed isn't a good plan at all.
so now you can leave it installed, and put HARDLINK="no" in your /etc/sysconfig/kernel
Dave
Once upon a time, Dave Jones davej@redhat.com said:
only if you have a huge number of kernels installed, which shouldn't be the case.
If you don't have a huge number of kernels installed, hardlink isn't a big benefit. The kernel-devel package is 14MB installed. Hardlinking with 3 kernel-devel packages installed will save no more than 28MB (and probably less). That is lost in the noise of a normal system. Just doing "yum update" will often use much more than 28MB; even after a "yum clean packages" on a rawhide system I see 90MB in /var/cache/yum.
Also, hardlink apparently may not be very efficient. I whipped up a version in perl that (in my limited testing) appears faster. Others have written C and C++ (and maybe IIRC python) versions that also appear to be faster. How about replacing it with something better performing?
The solution of "rpm -e hardlink" is also not very good, as some people use hardlink for other things (and so can't just remove it).
As a compromise, how about only kicking hardlink off if there are more than X kernel-devel packages installed (where X is more than the norm, like 5)? Or how about making the hardlink run as a cron job?
Another though: why the "find | while ; do hardlink"? You should just be able to do:
if [ -x /usr/sbin/hardlink ] ; then /usr/sbin/hardlink /usr/src/kernels/*FC* fi
The post script is forking hardlink over 5000 times as well as causing thousands of repeated directory lookups by find and the shell (that hardlink then has to do anyway), instead of just letting one run of hardlink do its job (it should recurse just fine).
On Tue, Jan 31, 2006 at 07:26:03PM -0600, Chris Adams wrote:
If you don't have a huge number of kernels installed, hardlink isn't a big benefit. The kernel-devel package is 14MB installed. Hardlinking with 3 kernel-devel packages installed will save no more than 28MB (and probably less). That is lost in the noise of a normal system. Just doing "yum update" will often use much more than 28MB; even after a "yum clean packages" on a rawhide system I see 90MB in /var/cache/yum.
"We can spew 28MB in app X because app Y spews 90MB" doesn't make a great deal of sense. The cumulative effects if every package took the same attitude would be a vastly bloated minimal install.
Also, hardlink apparently may not be very efficient. I whipped up a version in perl that (in my limited testing) appears faster. Others have written C and C++ (and maybe IIRC python) versions that also appear to be faster. How about replacing it with something better performing?
File a bug against it with patches, and I'm sure jnovy@redhat will be interested.
The solution of "rpm -e hardlink" is also not very good, as some people use hardlink for other things (and so can't just remove it).
true
As a compromise, how about only kicking hardlink off if there are more than X kernel-devel packages installed (where X is more than the norm, like 5)? Or how about making the hardlink run as a cron job?
I'll merge Ignacio's change to source /etc/sysconfig/kernel, in addition to the other change,
Another though: why the "find | while ; do hardlink"? You should just be able to do:
if [ -x /usr/sbin/hardlink ] ; then /usr/sbin/hardlink /usr/src/kernels/*FC* fi
The post script is forking hardlink over 5000 times as well as causing thousands of repeated directory lookups by find and the shell (that hardlink then has to do anyway), instead of just letting one run of hardlink do its job (it should recurse just fine).
I've often wondered why it was written that way too. Arjan, do you recall ?
Dave
On Tue, 2006-01-31 at 20:48 -0500, Dave Jones wrote:
On Tue, Jan 31, 2006 at 07:26:03PM -0600, Chris Adams wrote:
The post script is forking hardlink over 5000 times as well as causing thousands of repeated directory lookups by find and the shell (that hardlink then has to do anyway), instead of just letting one run of hardlink do its job (it should recurse just fine).
I've often wondered why it was written that way too. Arjan, do you recall ?
It does it so that you ensure that you're only hardlinking the same copy of any foo.h in the directory layout instead of all copies
Jeremy
Once upon a time, Jeremy Katz katzj@redhat.com said:
It does it so that you ensure that you're only hardlinking the same copy of any foo.h in the directory layout instead of all copies
Why? hardlink isn't going to replace files that are different, so there is no gain in not letting hardlink link all copies that are the same.
It is smart enough to look at the stat() info before reading the contents for comparison, so files of different sizes aren't going to be compared just because they have the same name.
The whole point of running hardlink is to reduce disk usage; why not let it try to do the best job possible?
On Tue, 2006-01-31 at 17:21 -0500, Dave Jones wrote:
On Tue, Jan 31, 2006 at 07:44:17PM +0100, Ralf Ertzinger wrote:
Hi.
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
only if you have a huge number of kernels installed, which shouldn't be the case.
This is not entirely true. I have a K6-2 500 with 256 M of memory. It takes FOREVER to install or remove a kernel when there are only 1 or 2 installed. It is about the only thing that takes forever.
Trever -- "What makes his world so hard to see clearly is not its strangeness but its usualness. Familiarity can blind you." -- Robert M. Pirsig
On Wed, Feb 01, 2006 at 08:57:37AM -0700, Trever L. Adams wrote:
On Tue, 2006-01-31 at 17:21 -0500, Dave Jones wrote:
On Tue, Jan 31, 2006 at 07:44:17PM +0100, Ralf Ertzinger wrote:
Hi.
I't like to request to make the hardlink stage kernel-devel does on every install optional. It takes a damn lot of time on my not-too-fast machines, I do not exactly care about the saved disk space, and given the rate of kernel changes in rawhide there are a lot of wasted CPU cycles there.
only if you have a huge number of kernels installed, which shouldn't be the case.
This is not entirely true. I have a K6-2 500 with 256 M of memory. It takes FOREVER to install or remove a kernel when there are only 1 or 2 installed. It is about the only thing that takes forever.
Removing hardlink will speed up installation/removal of 'kernel-devel', not 'kernel'. That's not going to get any faster, just due to the amount of data that has to be written.
Dave
On Wed, 2006-02-01 at 11:05 -0500, Dave Jones wrote:
On Wed, Feb 01, 2006 at 08:57:37AM -0700, Trever L. Adams wrote:
This is not entirely true. I have a K6-2 500 with 256 M of memory. It takes FOREVER to install or remove a kernel when there are only 1 or 2 installed. It is about the only thing that takes forever.
Removing hardlink will speed up installation/removal of 'kernel-devel', not 'kernel'. That's not going to get any faster, just due to the amount of data that has to be written.
Dave
Well, ps xa says that it is spending ALL of its time in hardlinking.
However, as you say...
Trever -- "God is real, unless declared integer." -- Unknown
On Wed, Feb 01, 2006 at 09:45:15AM -0700, Trever L. Adams wrote:
This is not entirely true. I have a K6-2 500 with 256 M of memory. It takes FOREVER to install or remove a kernel when there are only 1 or 2 installed. It is about the only thing that takes forever.
Removing hardlink will speed up installation/removal of 'kernel-devel', not 'kernel'. That's not going to get any faster, just due to the amount of data that has to be written.
Well, ps xa says that it is spending ALL of its time in hardlinking.
During -devel install yes, but you mentioned the actual 'kernel' package. The %post for the kernel doesn't do any hardlinking at all, and hasn't for a long time.
Dave
devel@lists.stg.fedoraproject.org
-
Andy Burns -
Chris Adams -
Dave Jones -
Ignacio Vazquez-Abrams -
Jeremy Katz -
Matthias Saou -
Paul Wouters -
Peter Jones -
Ralf Ertzinger -
Trever L. Adams