In general, what is the frequency that the fedora/redhat kernel folks grab updates from external source trees? I am mostly interested in how often it is done for JFFS2, but there are other trees (such as linuxppc) where development is done outside of the kernel.org tree that has a value to end users.
Some trees don't push to Linus very often (like JFFS2) for various reasons, so are users getting whatever is in the vanilla kernels? Or, do the fedora/redhat people do their own patches?
I am not expecting a schedule or some set update frequency. I am mostly just asking to see if it is done at all, and if so get a rough estimate as to how often.
thx, josh
I'm not a redhat/fedora developer - but my experience with past versions of red hat is that once a distro reaches release, they generally do not make any more changes to it other than bug fixes.
There is a very good reason for this - when you add new code to the kernel, you have a greater chance that something will break that someone depends upon. Especially the ability to compile a third party driver against the kernel source.
When FC2 is released, people who develop third party drivers will work to get their drivers building against the Fedora kernel source. If an update to the kernel changes something they depend upon, then Fedora users won't be able to build a new module until the third party updates their driver to match the changes made by Fedora.
You can get updated kernels with various cool things hacked into them from third parties, or even from development trees within Fedora, but you probably are only going to find bug fixes within the stable kernel for any given version.
I use to think that was a bad thing, but now I have come to realize it is a good thing.
On Mon, 2004-04-26 at 17:11, Josh Boyer wrote:
In general, what is the frequency that the fedora/redhat kernel folks grab updates from external source trees? I am mostly interested in how often it is done for JFFS2, but there are other trees (such as linuxppc) where development is done outside of the kernel.org tree that has a value to end users.
Some trees don't push to Linus very often (like JFFS2) for various reasons, so are users getting whatever is in the vanilla kernels? Or, do the fedora/redhat people do their own patches?
I am not expecting a schedule or some set update frequency. I am mostly just asking to see if it is done at all, and if so get a rough estimate as to how often.
thx, josh
On Mon, 2004-04-26 at 21:59, Michael A. Peters wrote:
There is a very good reason for this - when you add new code to the kernel, you have a greater chance that something will break that someone depends upon. Especially the ability to compile a third party driver against the kernel source. [...]
Yes, I know this and I understand the benefits. My question was more for the development kernels. I.e. before an official release, do the redhat/fedora kernels get stuff pulled in from other trees? I probably should have been more clear.
thx, josh
On Tue, 2004-04-27 at 06:14 -0500, Josh Boyer wrote:
On Mon, 2004-04-26 at 21:59, Michael A. Peters wrote:
There is a very good reason for this - when you add new code to the kernel, you have a greater chance that something will break that someone depends upon. Especially the ability to compile a third party driver against the kernel source. [...]
Yes, I know this and I understand the benefits. My question was more for the development kernels. I.e. before an official release, do the redhat/fedora kernels get stuff pulled in from other trees? I probably should have been more clear.
Not a RH kernel developer but .... I would be quite certain that no extra effort is made to pull in external trees just prior to release. That would invalidate all of the prior testing to ensure that a stable kernel is included in the release. Sure, you may lose some nifty new feature, or even miss out on a few bug fixes, but the end goal is a known commodity that can be unleashed on the world. The bug fixes can be incorporated into an errata kernel after they have been more thoroughly tested.
On Tue, 2004-04-27 at 13:16, David T Hollis wrote:
Not a RH kernel developer but .... I would be quite certain that no extra effort is made to pull in external trees just prior to release. That would invalidate all of the prior testing to ensure that a stable kernel is included in the release. Sure, you may lose some nifty new feature, or even miss out on a few bug fixes, but the end goal is a known commodity that can be unleashed on the world. The bug fixes can be incorporated into an errata kernel after they have been more thoroughly tested.
Yep, and I agree with all of that too. I guess I am still not being very clear, so here goes another attempt at asking this question:
At _any_ point during the development of the kernel for a new product release, do the kernel developers bring in changes from external trees? If so, which ones?
Obviously during development they pull from the kernel.org tree. For FC2, the kernel has gone from pre 2.6, to 2.6.5 already. It's pretty common knowledge that Red Hat/Fedora kernels contain changes by the kernel developers for various reasons (i.e. bug fixes, backports, etc). So, do those changes contain fixes from other trees or all they all done "in-house"?
If they do pull in changes from external trees, it might be easier to open bugs and point them to the tree for a fix. Or maybe I am just blabbering. I guess I am just curious.
Thanks for all the responses. They are appreciated even though I sound like a broken record :)
thx, josh
On Tue, 2004-04-27 at 15:17, Josh Boyer wrote:
At _any_ point during the development of the kernel for a new product release, do the kernel developers bring in changes from external trees? If so, which ones?
Yes they do. I know in the past red hat releases, they have backported stuff from the devel series kernel into the one they are shipping (IE backport 2.3 stuff into 2.2 - and 2.5 stuff into 2.6) and I believe they also do take some stuff from other sources as well.
On Tue, 2004-04-27 at 23:17, Josh Boyer wrote:
At _any_ point during the development of the kernel for a new product release, do the kernel developers bring in changes from external trees? If so, which ones?
It's something that we've done in the past which we're trying very hard not to do for FC2. (and we're doing a good job at sticking to that so far). A number of reasons for this..
- The 'lets stay close to mainline' mantra. - 'lets not end up with a fork of random trees that only ever lives on in a Red Hat tree'. - External trees often contain stuff that never makes it into mainline for good reason.
Obviously during development they pull from the kernel.org tree. For FC2, the kernel has gone from pre 2.6, to 2.6.5 already. It's pretty common knowledge that Red Hat/Fedora kernels contain changes by the kernel developers for various reasons (i.e. bug fixes, backports, etc). So, do those changes contain fixes from other trees or all they all done "in-house"?
Stuff in the current kernel falls into a few categories.. - Red Hat specific hacks (A lot less of these though these days, I think the noninterative oldconfig thing is all thats left) - New features we developed which upstream either isn't ready for, or won't take for 2.6, but folks repeatedly ask for (Exec-shield, 4G/4G, etc.) - Stuff that's destined for mainline real-soon-now. A few cherry picked bits of -mm, and bits and pieces developed by Red Hat folks. - Infrastructure work needed for other bits in FC (SELinux patches etc) These also are destined for upstream, but may block on other stuff going in first etc..
If they do pull in changes from external trees, it might be easier to open bugs and point them to the tree for a fix. Or maybe I am just blabbering. I guess I am just curious.
Pulling individual fixes is deemed the safer alternative, but even better is to get this stuff into mainline. Pressure the external tree maintainers to get their stuff merged. If it's a critical bug it's certainly something we'll consider adding to the tree until it gets fixed in mainline.
Thanks for all the responses. They are appreciated even though I sound like a broken record :)
Here's some fun stats with the number of patches against the current kernel I have checked out..
My current FC2 working tree from last weekend - 27 patches Fedora Core 1 - 102 patches RHL9 - 143 patches RHEL3 - 315 patches.
FC1 was off to a bad start with a pretty high patchcount from the beginning, and in a lot of cases, it's a real pain to maintain, so keeping this count as low as possible without sacrificing usability is a goal worthy of trying to maintain for as long as possible.
Having a heavily patched tree isn't necessarily a bad thing (ie, RHEL) as long as you have enough folks dedicated to maintaining it. We have that bodycount for RHEL3, we don't for FC. Keeping things as close to upstream also means that a lot of bugs we hit are likely reproducable upstream, and can end up getting fixed up by the upstream maintainer, which takes additional burden off of us, giving us more time to concentrate on other things, like finding new interesting ways to break the NVIDIA driver. j/k 8-)
By keeping patch count low, it also gives us extra incentive to push all the bits we come up with back upstream asap too.
Dave
On Wed, 28 Apr 2004, Dave Jones wrote:
Got my attention!
FC2, the kernel has gone from pre 2.6, to 2.6.5 already. It's pretty common knowledge that Red Hat/Fedora kernels contain changes by the kernel developers for various reasons (i.e. bug fixes, backports, etc). So, do those changes contain fixes from other trees or all they all done "in-house"?
Stuff in the current kernel falls into a few categories..
- Red Hat specific hacks (A lot less of these though these days, I think the noninterative oldconfig thing is all thats left)
- New features we developed which upstream either isn't ready for, or won't take for 2.6, but folks repeatedly ask for (Exec-shield, 4G/4G, etc.)
- Stuff that's destined for mainline real-soon-now. A few cherry picked bits of -mm, and bits and pieces developed by Red Hat folks.
- Infrastructure work needed for other bits in FC (SELinux patches etc) These also are destined for upstream, but may block on other stuff going in first etc..
Unfortuneatly, I have to agree with your goals. I've had pain due to the divergence of the RH kernels several times in the past.
If they do pull in changes from external trees, it might be easier to open bugs and point them to the tree for a fix. Or maybe I am just blabbering. I guess I am just curious.
Pulling individual fixes is deemed the safer alternative, but even better is to get this stuff into mainline. Pressure the external tree maintainers to get their stuff merged. If it's a critical bug it's certainly something we'll consider adding to the tree until it gets fixed in mainline.
But it's hard to get the attention of kernel tree maintainers. Often you never know if your patch "is not good enough" or "what may be needed" as no one gives it serious attention. Or it's just ignored over and over again until someone with influence notices and asks "is something wort while going on here". Next thing you get a mild caning for not developing "out of the tree".
OK so it's not your problem. I know.
And I don't have any ideas on how to improve the situation. With so much happening it must be very hard for the tree maintainers.
How bout you?
Here's some fun stats with the number of patches against the current kernel I have checked out..
My current FC2 working tree from last weekend - 27 patches Fedora Core 1 - 102 patches RHL9 - 143 patches RHEL3 - 315 patches.
Well done!
Ian
On Wed, 2004-04-28 at 10:15, raven@themaw.net wrote:
But it's hard to get the attention of kernel tree maintainers. Often you never know if your patch "is not good enough" or "what may be needed" as no one gives it serious attention. Or it's just ignored over and over again until someone with influence notices and asks "is something wort while going on here". Next thing you get a mild caning for not developing "out of the tree".
OK so it's not your problem. I know.
And I don't have any ideas on how to improve the situation. With so much happening it must be very hard for the tree maintainers.
How bout you?
Yeah, it is hard sometimes. The way I see it, you can do 2 things.
1) keep sending your patch(es) even if they get ignored at first if you really think you are right. Or
2) find one of those developers with influence and try getting it accepted through them
Personally, I like option 2. If I were a tree maintainer that had lots of patches coming in, I wouldn't have time to look at everything that Joe User sent me. By going through developers that the tree maintainer trusts, you spread the load maintaining the tree.
If you do it often enough and your patches are good, then usually the developer you are going through will start to mention that the fixes are coming from you. Look at some of the mainline kernel Changelogs. You see stuff like:
trini@kernel.crashing.org PPC32: More cleanups of the IBM Spruce code. From Randy Vinson rvinson@mvista.com.
If you get enough of those, you will eventually become trusted.
And please don't think I am presenting this as my idea. Obviously it's already being done. I was just putting in my $.02 since you asked :).
josh
On Wed, 2004-04-28 at 08:49, Dave Jones wrote:
It's something that we've done in the past which we're trying very hard not to do for FC2. (and we're doing a good job at sticking to that so far). A number of reasons for this..
Ah, now I am getting the answers I was looking for. Once again, Dave to the rescue :).
Stuff in the current kernel falls into a few categories..
- Red Hat specific hacks (A lot less of these though these days, I think the noninterative oldconfig thing is all thats left)
- New features we developed which upstream either isn't ready for, or won't take for 2.6, but folks repeatedly ask for (Exec-shield, 4G/4G, etc.)
- Stuff that's destined for mainline real-soon-now. A few cherry picked bits of -mm, and bits and pieces developed by Red Hat folks.
- Infrastructure work needed for other bits in FC (SELinux patches etc) These also are destined for upstream, but may block on other stuff going in first etc..
Ok, I figured that was the case.
Pulling individual fixes is deemed the safer alternative, but even better is to get this stuff into mainline. Pressure the external tree maintainers to get their stuff merged. If it's a critical bug it's certainly something we'll consider adding to the tree until it gets fixed in mainline.
Sometimes that is easier said than done, but you make a good point. Alot of the lag in merging an external tree is that it takes time, and not many people have an excess of that :).
Having a heavily patched tree isn't necessarily a bad thing (ie, RHEL) as long as you have enough folks dedicated to maintaining it. We have that bodycount for RHEL3, we don't for FC. Keeping things as close to upstream also means that a lot of bugs we hit are likely reproducable upstream, and can end up getting fixed up by the upstream maintainer, which takes additional burden off of us, giving us more time to concentrate on other things, like finding new interesting ways to break the NVIDIA driver. j/k 8-)
I suppose that the number of patches correlates with the number of features the customer requires too. And what those customers are willing to pay to get support for those features. Everyone knows the saying "You get what you paid for" ;).
By keeping patch count low, it also gives us extra incentive to push all the bits we come up with back upstream asap too.
I like the direction the FC2 kernel is going. Not only does it make things simpler for the reasons you stated, but it allows lowly kernel hackers like myself to see some of the focus points that Red Hat works on.
I am looking forward to when the Fedora CVS tree opens up. It'll be easier to come up with patches when you can diff against the current source, instead of an older RPM where you don't know if something has been fixed already or not. That's not to say that the kernel developers are necessarily looking forward to getting more patches from us, but at least we can try to help when needed :).
thx, josh
On Thu, 2004-04-29 at 01:06, Josh Boyer wrote:
I am looking forward to when the Fedora CVS tree opens up. It'll be easier to come up with patches when you can diff against the current source, instead of an older RPM where you don't know if something has been fixed already or not
Indeed, it will make life a lot easier for external contributors. Right now you have to pull down the whole whopping great SRPM each time. A 'cvs update' will save folks from having to do that. There's still a lot of stuff going on internally to try and make all of that happen. I wish we had something more than "stay tuned!" to say, but sadly, I don't think we do yet. Cristian and a bunch of others are working at making this happen asap.
That's not to say that the kernel developers are necessarily looking forward to getting more patches from us, but at least we can try to help when needed :).
On the contrary, if the patches are sane, I'm looking forward to it 8)
Dave
On Thu, 2004-04-29 at 14:31, Dave Jones wrote:
On Thu, 2004-04-29 at 01:06, Josh Boyer wrote:
I am looking forward to when the Fedora CVS tree opens up. It'll be easier to come up with patches when you can diff against the current source, instead of an older RPM where you don't know if something has been fixed already or not
Indeed, it will make life a lot easier for external contributors. Right now you have to pull down the whole whopping great SRPM each time. A 'cvs update' will save folks from having to do that.
it is trivial and if people are interested, I can make the expanded src.rpm available on people.redhat.com.
That's not to say that the kernel developers are necessarily looking forward to getting more patches from us, but at least we can try to help when needed :).
On the contrary, if the patches are sane, I'm looking forward to it 8)
But to be honest, the best way to get kernel changes done is to send them for inclusion into the upstream (kernel.org) kernels. The first question I and Dave will ask for *any* patch is "why isn't it in upstream", because if it's not good enough for upstream why would it be good enough for us?
On Thu, 2004-04-29 at 07:38, Arjan van de Ven wrote:
But to be honest, the best way to get kernel changes done is to send them for inclusion into the upstream (kernel.org) kernels. The first question I and Dave will ask for *any* patch is "why isn't it in upstream", because if it's not good enough for upstream why would it be good enough for us?
Good point. But that also kinda goes back to the issue that was brought up in this thread about it being difficult sometimes to get patches merged because we don't have immediate name recognition.
Don't get me wrong, I think that folks should definitely send patches to the mainline first. It's just that sometimes we may need a little help getting those patches looked at. So if we send them to both Fedora and the mainline, maybe someone in Fedora will point out in the merits of the patch (or lack there of) to Linus and company. Or maybe not :).
josh
On Thu, Apr 29, 2004 at 01:31:24PM +0100, Dave Jones wrote:
Indeed, it will make life a lot easier for external contributors. Right now you have to pull down the whole whopping great SRPM each time. A 'cvs update' will save folks from having to do that.
I am in firm agreement here. Early in the FC2 cycle, I was very interested in working with the kernel. I came to realise that everything I was looking at was at least 36 hours old (dave -> buildsystem, buildsystem -> rawhide, rawhide -> mirror, mirror -> local sync) In the end I just gave up for this test cycle because 36 hours is a very long time that early in the test release process. Hopefully CVS will come online at some point and ease the lag there, better if it is in time for the FC3 test cycle.
Justin
On Thu, 2004-04-29 at 17:07, Justin M. Forbes wrote:
On Thu, Apr 29, 2004 at 01:31:24PM +0100, Dave Jones wrote:
Indeed, it will make life a lot easier for external contributors. Right now you have to pull down the whole whopping great SRPM each time. A 'cvs update' will save folks from having to do that.
I am in firm agreement here. Early in the FC2 cycle, I was very interested in working with the kernel. I came to realise that everything I was looking at was at least 36 hours old (dave -> buildsystem,
normally this is then followed by an immediate rsync to http://people.redhat.com/arjanv/2.6 to take out the time it takes for rawhide to sync
Le mar 27/04/2004 à 02:11, Josh Boyer a écrit :
In general, what is the frequency that the fedora/redhat kernel folks grab updates from external source trees? I am mostly interested in how often it is done for JFFS2, but there are other trees (such as linuxppc) where development is done outside of the kernel.org tree that has a value to end users.
I am running Fedora Core 1 on a hush Epia-M10000 machine. I _HAD_ to roll my own kernel with video driver patches:
- the patches applied to the Fedora kernel src rpm weren't working
- I wanted to benefit from the 2.6.x kernel speed increase, on a 1Ghz machine any speed increse is more important than on a 2.x Ghz machine.
The video stuff is stable and performing quite well so my next update of kernel will be when I change to FC 2.
Hope that helps
Tony Grant
On Tue, 2004-04-27 at 00:20, Tony Grant wrote:
I am running Fedora Core 1 on a hush Epia-M10000 machine. I _HAD_ to roll my own kernel with video driver patches:
the patches applied to the Fedora kernel src rpm weren't working
I wanted to benefit from the 2.6.x kernel speed increase, on a 1Ghz
machine any speed increse is more important than on a 2.x Ghz machine.
The video stuff is stable and performing quite well so my next update of kernel will be when I change to FC 2.
Hope that helps
I do the same with JFFS2. dwmw2 has a wonderful script that patches the current JFFS2 tree into almost any recent 2.4 or 2.6 kernel. I was just curious how often this is done by the official kernel developers _before_ an official release (i.e. the rawhide kernels).
thx, josh