On Sat, Jul 5, 2008 at 5:14 PM, Thorsten Leemhuis fedora@leemhuis.info wrote:

On 05.07.2008 15:54, drago01 wrote:

...

On Sat, Jul 5, 2008 at 2:56 PM, Thorsten Leemhuis fedora@leemhuis.info wrote:

...

• a karama of "+3" in bodhi seems not enough for a auto-move from testing

to stable (or even worse: straight to stable if enough people tested the kernel and gave their +1 after the update got filed in bodhi but *before* it actually hit fedora-testing) if there are no other pressing issues (like security fixes). The kernel is a to complex beast; more then 3 people should be needed to give a +1. And a bit of time needs to pass to give enough people the opportunity to install, test and report problems with new kernels.

Well the problem is not the patches that are being shipped but bodhi.

Yes and no. The patches are quite big and carry a additional risk. We don't take such risk in other areas (Sound, LAN, Storage -- there for similar reasons it might make sense) -- so why should we take that risk for WLAN drivers in stable releases (might be something else for rawhide now and then)?

There was a reasons until now (upstream sucked until a few months ago), but we IMHO have to stop that sooner or later (otherwise Alsa maintainers, Jeff G./Alan Cox might want to do the same and then it really becomes problematic). As the most important WLAN bits are in the kernel now with 2.6.26 it's IMHO a good time to think about slowing down a bit. Of cause we can still cherry picking some improvements if we want.

Well if the upstream maintainer sees a need for this why not? (given the changes go to testing first)

...

Auto pushing for something like the kernel should be disabled, to prevent such stuff from happening. The bug you are referring to, has been resolved quickly, if the kernel stayed in testing (ie no autopush) it would not have hit stable with this bug.(same for other, non wireless related issues).

Well, that is round about what I said in my discussion point just in slightly different words ;-)

Well this is because we agree here ;)

On Sat, Jul 05, 2008 at 02:56:30PM +0200, Thorsten Leemhuis wrote:

John (CCed), I really appreciate your work in the wireless area and would like to use the opportunity to say "thanks for all you work", as support for WLAN hardware in the Linux kernel improved a lot in the upstream kernel and Fedora thx to your (and other linux wireless developers) work over the last two years.

Thanks. Normally my reward is to be kicked in the teeth when something breaks -- usually by the same people who would be screaming "but this is already fixed upstream" if I didn't push very recent patches, but I digress...

But we now for at least the second time in the past few weeks had/have a more-than-minor wireless breakage in a Fedora kernel for a released distro (bug #453390 now; http://lwn.net/Articles/286558/ is discussing the one some weeks ago; I think there was one more breakage not that long ago, but I can't remember). I and many users (see for example #453390) got hit by those problems. That's why I was wondering: what are we at Fedora doing to prevent similar problems in the future?

For the record, bug 453390 was caused by me screwing-up a fixup for build breakage between the current wireless code and the base 2.6.25 kernel in F9. In other words, this was not the result of merging some buggy upstream patch. Instead it was the result of simple human error on my part. I would also point-out that cherry-picking individual fixes often means rebasing that fix between upstream and the target kernel, and doing so creates an opportunity for such "human error" mistakes to creep-in _with_every_single_fix_.

If we want to cherry-pick individual wireless LAN fixes into Fedora kernels then we need another monkey. I already spend more time than I should spare keeping Fedora kernels up-to-date as it is. But all the smiling faces are my reward...

Three things spring to my mind and I just propose then here for discussion; maybe something good comes out of it in the end:

• a karama of "+3" in bodhi seems not enough for a auto-move from

testing to stable (or even worse: straight to stable if enough people tested the kernel and gave their +1 after the update got filed in bodhi but *before* it actually hit fedora-testing) if there are no other pressing issues (like security fixes). The kernel is a to complex beast; more then 3 people should be needed to give a +1. And a bit of time needs to pass to give enough people the opportunity to install, test and report problems with new kernels. For the latest kernel it seems to me that "to less time" really was the problem, otherwise the problem from #453390 would have been noticed earlier

Something is definitely broken here. I seem to recall beating the drum for Karma in the not-too-distant past, when the required number seemed to be up in the teens? Who's bright idea was it to bring this value down to +3? My assumption had been that it was okay to push these wireless bits because Bodhi would keep us from releasing truly broken kernels. If we are going to use +3 then my assumption is clearly wrong and my practices have to change.

• should we separate security updates and other kernel fixes in a better

way to make sure those "other fixes" get proper testing before they get send out to the users?

Sounds good, but I have no idea how to do that. Does Fedora need a "z-stream" a la RHEL?

• John, having all those pending and not-yet-upstream-merged

improvements for wireless hardware in the Fedora kernel was something good in the past when WLAN support in the kernel was quite bad/incomplete. But the main and most important bits for proper wireless hardware support seem to be in the upstream kernel now; sure, there will always be improvements in the queue, but that's the same in most other linux subsystems with drivers as well. So I'm wondering: isn't it time now to finally stop shipping all those wireless-next bits (currently quite some big patches; see: -rw-rw-r-- 1 thl thl 2484 14. Mär 17:06 linux-2.6-ms-wireless-receiver.patch -rw-rw-r-- 1 thl thl 39874 4. Jul 22:21 linux-2.6-wireless-fixups.patch -rw-rw-r-- 1 thl thl 2656652 4. Jul 22:21 linux-2.6-wireless-pending.patch -rw-rw-r-- 1 thl thl 4165718 4. Jul 22:21 linux-2.6-wireless.patch ) in released Fedora Version (e.g. 8 and 9 currently) when we start shipping 2.6.26?

Perhaps it is worth explaing a bit about what these are:

-- linux-2.6-wireless.patch contains the stream of wireless patches going from the base kernel's release (currently 2.6.25) and the next upstream release (currently 2.6.26);

-- linux-2.6-wireless-pending.patch contains the stream of wireless patches from the next upstream release to the following release (currently 2.6.27);

-- linux-2.6-wireless-fixups.patch contains the changes required to un-break the build after applying the previous two patches (a screw-up here caused bug 453390).

So if you dropped the -wireless patch (i.e. the current contents of the -pending patch) when F9 picks-up 2.6.26, you would be going backwards in your wireless support.

FWIW, this practice started in Rawhide ages ago. Then it continued into F7 or so because wireless continued to lag expectations. But at some point things got enough better for enough people that now my judgment is widely questioned. So I've been thinking for some time that the process should change, but I was looking for a graceful transition.

So, here is what I propose:

-- continue the current practice until 2.6.26 is released and the 2.6.27 merge window closes;

-- after that, continue the current practice for updating rawhide; but,

-- once F9 (and presumably F8) move to 2.6.26, move the -pending bits to the -wireless.patch and do _not_ create a new -pending.patch for 2.6.28 bits;

-- once 2.6.27 is released, drop the -wireless patch and F9/F8 will get no more wireless updates at all;

-- F10 will release with -wireless and -pending patches inherited from rawhide, but they will "age out" following the process described for F9/F8 above.

The alternative would be to simply remove the -pending (and possibly -wireless) patches from F10 early after it is cut, but that seems jittery to me.

I will defer to the judgment of the group -- as I've said I spend a lot more time keeping Fedora up-to-date than I would like to be doing. Just don't expect me to transfer that effort over to tireless cherry-picking of fixes, for I just do not have the time.

Thanks,

John

P.S. This quote is based on ignorance by the person being quoted:

On Sat, Jul 05, 2008 at 05:50:52PM +0200, Thorsten Leemhuis wrote:

Or, to abuse some words from someone else in the discussions around separately packaged kernel modules for Fedora: "If they [the patches in this case] are not good enough to get applied upstream why should they be good enough for us?" There is a reason for the short merge window and the longer stabilization phase upstream.

None of the patches in question meet this definition. They are all queued for upstream. Some of them are queued for the next upstream release rather than the current one due mostly to the upstream release process's scheduling requirements. But they are all upstream material. The only patch I've added to Fedora that does not meet this definition is the one for the at76_usb driver, which is a special case. I'm happy to discuss at76_usb separately if someone so wishes.

John

On Mon, 2008-07-07 at 11:37 -0400, John W. Linville wrote:

So, here is what I propose: -- continue the current practice until 2.6.26 is released and the 2.6.27 merge window closes;

-- after that, continue the current practice for updating rawhide; but,

Definitely makes good sense for rawhide. We may want to decide to go to the "released" practice at some point late in the cycle, but that's fine-tuning probably

-- once F9 (and presumably F8) move to 2.6.26, move the -pending bits to the -wireless.patch and do _not_ create a new -pending.patch for 2.6.28 bits; -- once 2.6.27 is released, drop the -wireless patch and F9/F8 will get no more wireless updates at all;

This sounds reasonable. And to be fair, they will get wireless updates, but only when they get other drivers (eg, by going to 2.6.28) or things that are small and pushed to -stable upstream

-- F10 will release with -wireless and -pending patches inherited from rawhide, but they will "age out" following the process described for F9/F8 above.

Yep.

I will defer to the judgment of the group -- as I've said I spend a lot more time keeping Fedora up-to-date than I would like to be doing. Just don't expect me to transfer that effort over to tireless cherry-picking of fixes, for I just do not have the time.

I like the proposal and think it makes for a good balance -- we test new stuff to go into new releases and then the filtering process for older releases is based on when things make it upstream.

Jeremy

On 07.07.2008 18:51, Dave Jones wrote:

On Mon, Jul 07, 2008 at 12:30:28PM -0400, Jeremy Katz wrote:

...

...

-- once F9 (and presumably F8) move to 2.6.26, move the -pending bits to the -wireless.patch and do _not_ create a new -pending.patch for 2.6.28 bits; -- once 2.6.27 is released, drop the -wireless patch and F9/F8 will get no more wireless updates at all;

This sounds reasonable. And to be fair, they will get wireless updates, but only when they get other drivers (eg, by going to 2.6.28) or things that are small and pushed to -stable upstream

Yeah, this proposal makes a lot of sense to me.

+1, as that's round about one of the things I suggested in my mail ;-)

Just one additional comment to the mail from John:

...

...

Or, to abuse some words from someone else in the discussions around separately packaged kernel modules for Fedora: "If they [the patches in this case] are not good enough to get applied upstream why should they be good enough for us?" There is a reason for the short merge window and the longer stabilization phase upstream.

None of the patches in question meet this definition.

We misunderstood us here. ;-)

They are all queued for upstream. Some of them are queued for the next upstream release rather than the current one due mostly to the upstream release process's scheduling requirements. [...]

And that is actually what I meant. Upstream doesn't take them *now*, as it's considered to risky for upstream now, as upstream has a stabilization phase. So if it's to risky for them, then it's IMHO to risky for us in a stable update as well.

CU knurd

P.S.: For those interested, David Nielsen has a blog entry related to this discussion:

http://davidnielsen.wordpress.com/2008/07/06/pushing-kernels-more-aggressive...

David Nielsen: Pushing kernels more aggressively to updates-testing

On thing struck me tonight about the recent fiasco relating to the stable marking of a kernel that just happened to also kill wifi for a great number of users. We did the correct thing, to a degree naturally, the update was in relation to a security update something Fedora takes very seriously. As such our users should always feel safe knowing that we will push such updates fast, keeping their systems secure through multiple means including proactive security and rapid updates.

However the problem is that we don’t apply the update to the existing stable kernel, the patch is always applied on top of the progressing kernel, meaning we also end up shipping a lot of other things such as bugfixes, updates to the latest upstream STABLE tree and various other things. This however is confronted with one problem, the kernels in between the current stable and next update are not all being pushed to updates-testing - only selected kernel updates are. In cases where we then have to release a security fix we are forced to ship a bunch of stuff additionally which is not likely to have been tested extensively.

It occures to me that catching these bugs before they become a problem for average users could be accompliced by making better use of updates-testing, testers are normally willing to experience a degree of breakage and are qualified to file bugs for the most part. Then at least when an urgent update is required we will not likely be surprised by massive unrelated breakage - it might still occure but we can warn people if avoiding massive breakage is impossible and reverting the offending patches is impossible prior to release.

An additional problem caused by this is that when an urgent release contains bugs we will be urged to ship another update straight afterwards. Opening us to even more bugs from another untested delta (since other development is likely to have gone on along side the bugfix) and having our users suck down a second kernel package shortly after the original update.

The other option would be applying the security update to the current stable kernel and not carrying the current delta in the update, but this is expensive in terms of man power and time, it also goes counter to the rapidly developing nature of Fedora in general. This is the realm of the enterprise distros, if people want this approach something like RHEL/CentOS is likely a better fit for them.

John W. Linville wrote:

I still think that for continuity's sake f8 and f9 should continue to get wireless fixes from 2.6.27. Those should only be specific bug fixes (although I suppose there is still time for a new driver), so hopefully the nattering nabobs won't be opposed to continuing with that part of the original plan.

If you push wireless fixes to -stable Fedora would then get them for "free". I don't see many wireless patches in there generally, though the ath5k memory corruption fix just went in.

And new drivers would be great. Lots of people seem to want ath9k, for example.

On Tue, Aug 05, 2008 at 05:05:33PM -0400, Chuck Ebbert wrote:

John W. Linville wrote:

...

I still think that for continuity's sake f8 and f9 should continue to get wireless fixes from 2.6.27. Those should only be specific bug fixes (although I suppose there is still time for a new driver), so hopefully the nattering nabobs won't be opposed to continuing with that part of the original plan.

If you push wireless fixes to -stable Fedora would then get them for "free". I don't see many wireless patches in there generally, though the ath5k memory corruption fix just went in.

Well I see where you had to revert a few in the F-9 changelogs as you rebased on later -stable kernels, so there must be a few getting through. :-)

If you would like to nominate more wireless fixes for -stable, feel free to do so. As it is, I mostly rely on my driver/stack maintainers to identify appropriate patches for stable -- they are aware of the process, including the Cc: stable@kernel.org trick.

Even so, that only works for those problems which date back to 2.6.26. Since Fedora kernels already have 2.6.27 code, then the 2.6.27 patches would seem appropriate for Fedora too. Of course you could elide the 2.6.27 code from the Fedora kernels as you have suggested elsewhere, but then you will reintroduce other problems.

And new drivers would be great. Lots of people seem to want ath9k, for example.

We'll see. Honestly I've been dropping other things in favor of keeping Fedora up-to-date for the last year or more. Rather than find ways to spend just as much time on Fedora while "getting less bang for my buck", I'll probably find something else to occupy more of my time. In particular, I think ath9k will make it into the 2.6.27 queue.

John