Hello,
Over the past week, we've been dealing with a kernel bug[1] that prevents i686 machines from booting. Help was requested and given, and it has been excellent and most welcome. This email has no reflection on that, and is instead focused on the reality of where i686 stands today.
In February[2] we sent out an email highlighting that the kernel team was not going to treat i686 bugs as a priority. Since that time, we have held true to our word and have not focused on fixing i686 bugs at all. It seems that the wider community is also treating i686 similarly. The kernel bug that was made automatic blocker because of existing criteria was present in Fedora since the 4.1-rc6 kernel, which was released May 31. It has been in every boot.iso created since that date. Not a single person reported this issue until last week. That is a timespan of two months.
The kernel team has autotesting for i686 kernels, but the environment there does not utilize boot.iso so it did not detect this. The QA team has automated testing for some of this, but nothing for the i686 architecture at all. It is not a priority there either.
Perhaps it is time that we evaluate where i686 stands in Fedora more closely. For a starting suggestion, I would recommend that we do not treat it as a release blocking architecture. This is not the same as demotion to secondary architecture status. That has broader implications in both buildsys and ecosystem. My suggestion is narrowly focused so that builds still proceed as today, but if there is something broken for i686 it does not block the release of whatever milestone we are pursuing.
(To be clear, I would support a move to secondary arch status for i686, but I am not suggesting it at this time.)
Making i686 non-release blocking would actually match reality. None of the Fedora Editions appear at all concerned with i686. Cloud is demoting[3] i686 from its offering. Workstation has been fairly ambivalent about it and recommends x86_64. Server does the same. Given the lack of focus on it, and the fact that the broader community is not testing the development releases for i686, I believe this would be a good first step.
josh
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1247382 [2] https://lists.fedoraproject.org/pipermail/devel/2015-February/208368.html [3] https://fedorahosted.org/cloud/ticket/106
On Tue, Aug 04, 2015 at 09:47:27AM -0400, Josh Boyer wrote: [...snip...]
Perhaps it is time that we evaluate where i686 stands in Fedora more closely. For a starting suggestion, I would recommend that we do not treat it as a release blocking architecture. This is not the same as demotion to secondary architecture status. That has broader implications in both buildsys and ecosystem. My suggestion is narrowly focused so that builds still proceed as today, but if there is something broken for i686 it does not block the release of whatever milestone we are pursuing.
(To be clear, I would support a move to secondary arch status for i686, but I am not suggesting it at this time.)
So to put a finer point on this, our shipping i686 images depends on a broader community effort beyond the kernel maintainers in the Fedora Engineering team. That needs to precisely not mean more heroics on the part of e.g. QE, rel-eng, etc. I have no idea what the pushback on this issue is, but I'm sure this thread will tell us. But given that Fedora is supposed to encourage such community effort, it would be good to see what people are willing to do to build it.
Making i686 non-release blocking would actually match reality. None of the Fedora Editions appear at all concerned with i686. Cloud is demoting[3] i686 from its offering. Workstation has been fairly ambivalent about it and recommends x86_64. Server does the same. Given the lack of focus on it, and the fact that the broader community is not testing the development releases for i686, I believe this would be a good first step.
"Ambivalent" is probably understated here. It's hard to imagine people securing i686 hardware these days to run a Workstation experience, after all.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1247382 [2] https://lists.fedoraproject.org/pipermail/devel/2015-February/208368.html [3] https://fedorahosted.org/cloud/ticket/106
On Tue, Aug 04, 2015 at 10:40:28AM -0400, Paul W. Frields wrote:
"Ambivalent" is probably understated here. It's hard to imagine people securing i686 hardware these days to run a Workstation experience, after all.
The question, I think, is how much we want to prioritize the "Workstation experience" on older hardware (or on devices like the Baytrail tablets). When Owen tested this a year or so ago, the memory savings on 32-bit in Workstation were very significant, such that — kernel bugs aside — is clearly advantageous for systems with less than 3GB of RAM (and probably all the way up to 4GB).
I think it's perfectly fine for Workstation to acknowledge this and move on anyway — for those cases, there *is* more to Fedora, after all, and it may be that an overall-lightweight desktop environment is a better choice, and that's fine. The whole idea here is that any one product/edition/flavor/spin *doesn't* have to be all things to all people.
On Tue, 2015-08-04 at 10:47 -0400, Matthew Miller wrote:
On Tue, Aug 04, 2015 at 10:40:28AM -0400, Paul W. Frields wrote:
"Ambivalent" is probably understated here. It's hard to imagine people securing i686 hardware these days to run a Workstation experience, after all.
The question, I think, is how much we want to prioritize the "Workstation experience" on older hardware (or on devices like the Baytrail tablets).
Just to this point - if we wanted to support the Baytrail tablets properly we should probably get 64-on-32 working. Allowing 32-bit UEFI installs probably isn't something we want to do officially. The way Fedlet is built is honestly just the way that was easiest for me to hack up.
On 08/13/2015 03:17 PM, Adam Williamson wrote:
On Tue, 2015-08-04 at 10:47 -0400, Matthew Miller wrote:
On Tue, Aug 04, 2015 at 10:40:28AM -0400, Paul W. Frields wrote:
"Ambivalent" is probably understated here. It's hard to imagine people securing i686 hardware these days to run a Workstation experience, after all.
The question, I think, is how much we want to prioritize the "Workstation experience" on older hardware (or on devices like the Baytrail tablets).
Just to this point - if we wanted to support the Baytrail tablets properly we should probably get 64-on-32 working. Allowing 32-bit UEFI installs probably isn't something we want to do officially.
Has this changed due to IoT?
Thanks, Florian
On Apr 23, 2016 09:18, "Florian Weimer" fweimer@redhat.com wrote:
On 08/13/2015 03:17 PM, Adam Williamson wrote:
On Tue, 2015-08-04 at 10:47 -0400, Matthew Miller wrote:
On Tue, Aug 04, 2015 at 10:40:28AM -0400, Paul W. Frields wrote:
"Ambivalent" is probably understated here. It's hard to imagine people securing i686 hardware these days to run a Workstation experience, after all.
The question, I think, is how much we want to prioritize the "Workstation experience" on older hardware (or on devices like the Baytrail tablets).
Just to this point - if we wanted to support the Baytrail tablets properly we should probably get 64-on-32 working. Allowing 32-bit UEFI installs probably isn't something we want to do officially.
Has this changed due to IoT?
I am not sure there has been a very large amount of people interested in doing the work or looking at IoT on i386 since the majority of the hardware is arm and has no eufi
Thanks, Florian -- devel mailing list devel@lists.fedoraproject.org http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org
On Sat, 2016-04-23 at 09:27 -0600, Stephen John Smoogen wrote:
Just to this point - if we wanted to support the Baytrail tablets properly we should probably get 64-on-32 working. Allowing 32-bit
UEFI
installs probably isn't something we want to do officially.
Has this changed due to IoT?
I am not sure there has been a very large amount of people interested in doing the work or looking at IoT on i386 since the majority of the hardware is arm and has no eufi
I haven't looked at it at all, and haven't really had time for fedlet lately.
On 04/23/2016 05:27 PM, Stephen John Smoogen wrote:
On Apr 23, 2016 09:18, "Florian Weimer" fweimer@redhat.com wrote:
On 08/13/2015 03:17 PM, Adam Williamson wrote:
On Tue, 2015-08-04 at 10:47 -0400, Matthew Miller wrote:
On Tue, Aug 04, 2015 at 10:40:28AM -0400, Paul W. Frields wrote:
"Ambivalent" is probably understated here. It's hard to imagine people securing i686 hardware these days to run a Workstation experience, after all.
The question, I think, is how much we want to prioritize the "Workstation experience" on older hardware (or on devices like the Baytrail tablets).
Just to this point - if we wanted to support the Baytrail tablets properly we should probably get 64-on-32 working. Allowing 32-bit UEFI installs probably isn't something we want to do officially.
Has this changed due to IoT?
I am not sure there has been a very large amount of people interested in doing the work or looking at IoT on i386 since the majority of the hardware is arm and has no eufi
It's not about i386, it's about booting x86_64 systems from a 32-bit UEFI firmware. Apparently this is still a thing with Baytrail systems.
(It's not related to fedlet AFAICS, whatever that is.)
Florian
On Sat, 2016-04-23 at 19:54 +0200, Florian Weimer wrote:
I am not sure there has been a very large amount of people interested in doing the work or looking at IoT on i386 since the majority of the hardware is arm and has no eufi
It's not about i386, it's about booting x86_64 systems from a 32-bit UEFI firmware. Apparently this is still a thing with Baytrail systems.
(It's not related to fedlet AFAICS, whatever that is.)
fedlet is/was my project for running Fedora on Baytrail tablets, so yes, it's entirely related.
Paul W. Frields (stickster@gmail.com) said:
On Tue, Aug 04, 2015 at 09:47:27AM -0400, Josh Boyer wrote: [...snip...]
Perhaps it is time that we evaluate where i686 stands in Fedora more closely. For a starting suggestion, I would recommend that we do not treat it as a release blocking architecture. This is not the same as demotion to secondary architecture status. That has broader implications in both buildsys and ecosystem. My suggestion is narrowly focused so that builds still proceed as today, but if there is something broken for i686 it does not block the release of whatever milestone we are pursuing.
(To be clear, I would support a move to secondary arch status for i686, but I am not suggesting it at this time.)
So to put a finer point on this, our shipping i686 images depends on a broader community effort beyond the kernel maintainers in the Fedora Engineering team. That needs to precisely not mean more heroics on the part of e.g. QE, rel-eng, etc. I have no idea what the pushback on this issue is, but I'm sure this thread will tell us. But given that Fedora is supposed to encourage such community effort, it would be good to see what people are willing to do to build it.
Here's my perspective as an i686 Fedora user...
I have a box (2009-ish) that's in use as a file/backup server. As such, I don't spend a lot of time futzing with it - it doesn't run rawhide, it rarely runs the prereleases until beta or later time. If something breaks, I'll look at it, send some feedback, update it as necessary, and back off to a working version. And historically, it *hasn't* broken.
But, if it did break that hard... would I spend a month digging into the kernel source and bisecting to try and find a fix? Or would I spend the $100-120 to slap a new motherboard in it and install the x86_64 version?
I'd like to say I'd do the former. But realisitically it's the latter. And I wonder how much of the i686 Fedora-using community is in the same boat.
Bill
Perhaps it is time that we evaluate where i686 stands in Fedora more closely. For a starting suggestion, I would recommend that we do not treat it as a release blocking architecture. This is not the same as demotion to secondary architecture status. That has broader implications in both buildsys and ecosystem. My suggestion is narrowly focused so that builds still proceed as today, but if there is something broken for i686 it does not block the release of whatever milestone we are pursuing.
(To be clear, I would support a move to secondary arch status for i686, but I am not suggesting it at this time.)
So to put a finer point on this, our shipping i686 images depends on a broader community effort beyond the kernel maintainers in the Fedora Engineering team. That needs to precisely not mean more heroics on the part of e.g. QE, rel-eng, etc. I have no idea what the pushback on this issue is, but I'm sure this thread will tell us. But given that Fedora is supposed to encourage such community effort, it would be good to see what people are willing to do to build it.
Here's my perspective as an i686 Fedora user...
I have a box (2009-ish) that's in use as a file/backup server. As such, I don't spend a lot of time futzing with it - it doesn't run rawhide, it rarely runs the prereleases until beta or later time. If something breaks, I'll look at it, send some feedback, update it as necessary, and back off to a working version. And historically, it *hasn't* broken.
But, if it did break that hard... would I spend a month digging into the kernel source and bisecting to try and find a fix? Or would I spend the $100-120 to slap a new motherboard in it and install the x86_64 version?
I'd like to say I'd do the former. But realisitically it's the latter. And I wonder how much of the i686 Fedora-using community is in the same boat.
A lot of the users of i686 that I know use it from live images or installing live images which, and I've not followed the issue too closely so might be a little off here, wouldn't have hit the bug that was being seen by the installer side of things. All those uses would also generally not be a rawhide/dev branch user.
Peter
On 08/04/2015 08:38 AM, Peter Robinson wrote:
A lot of the users of i686 that I know use it from live images or installing live images which, and I've not followed the issue too closely so might be a little off here, wouldn't have hit the bug that was being seen by the installer side of things. All those uses would also generally not be a rawhide/dev branch user.
I maintain a bunch of P4 (non 64-bit) computers with max 2GB RAM at a school. I install them via PXE so don't directly deal with the install images. I have some spares now so I could bring one home to start doing some pre-release testing if that would help.
On 08/04/2015 05:12 PM, Bill Nottingham wrote:
Paul W. Frields (stickster@gmail.com) said:
Here's my perspective as an i686 Fedora user...
I have a box (2009-ish) that's in use as a file/backup server.
I have 3 i686 boxen.
2 are 2009-ish atom-netbook, one is a 2000-ish PIII-desktop.
As such, I don't spend a lot of time futzing with it - it doesn't run rawhide, it rarely runs the prereleases until beta or later time. If something breaks, I'll look at it, send some feedback, update it as necessary, and back off to a working version. And historically, it *hasn't* broken.
But, if it did break that hard... would I spend a month digging into the kernel source and bisecting to try and find a fix? Or would I spend the $100-120 to slap a new motherboard in it and install the x86_64 version?
I'd like to say I'd do the former. But realisitically it's the latter. And I wonder how much of the i686 Fedora-using community is in the same boat.
ACK. I would switch the 2009-atoms to Windows (They are dual boot with Win) and the PIII to a different Linux distro.
Ralf
On Tue, 2015-08-04 at 11:12 -0400, Bill Nottingham wrote:
Here's my perspective as an i686 Fedora user...
I have a box (2009-ish) that's in use as a file/backup server. As such, I don't spend a lot of time futzing with it - it doesn't run rawhide, it rarely runs the prereleases until beta or later time. If something breaks, I'll look at it, send some feedback, update it as necessary, and back off to a working version. And historically, it *hasn't* broken.
But, if it did break that hard... would I spend a month digging into the kernel source and bisecting to try and find a fix? Or would I spend the $100-120 to slap a new motherboard in it and install the x86_64 version?
I'd like to say I'd do the former. But realisitically it's the latter. And I wonder how much of the i686 Fedora-using community is in the same boat.
So we have a product that is installed on about ~80 netbooks running i386-PAE kernels. They are now running f21 I think. I considered updating them but they are offline machines for nearly their entire life. I had contemplated putting CentOS 7 but there is no i386 for that. I would imagine that the hardware would be replaced by newer netbooks that handle x64. If they can't be they'll run EOL'd versions of fedora till death. If I can update them eventually it wouldn't matter to me if the i386 system saw less love but eventually came out. Granted if fedora dropped i386 completely I'd find a distro to use that supported it I guess if any. It wouldn't be the end of the world for me.
On Tue, Aug 04, 2015 at 10:40:28 -0400, "Paul W. Frields" stickster@gmail.com wrote:
"Ambivalent" is probably understated here. It's hard to imagine people securing i686 hardware these days to run a Workstation experience, after all.
I still use i686 for my primary server, primary desktop and primary laptop. I just don't hit install issues as I rarely due fresh install on those machines. I do try to help debug kernel issues. I am unlikely to buy i686 hardware, but I might scrounge some if I could get it for free.
On Aug 4, 2015 9:40 AM, "Paul W. Frields" stickster@gmail.com wrote:
On Tue, Aug 04, 2015 at 09:47:27AM -0400, Josh Boyer wrote: [...snip...]
Perhaps it is time that we evaluate where i686 stands in Fedora more closely. For a starting suggestion, I would recommend that we do not treat it as a release blocking architecture. This is not the same as demotion to secondary architecture status. That has broader implications in both buildsys and ecosystem. My suggestion is narrowly focused so that builds still proceed as today, but if there is something broken for i686 it does not block the release of whatever milestone we are pursuing.
(To be clear, I would support a move to secondary arch status for i686, but I am not suggesting it at this time.)
So to put a finer point on this, our shipping i686 images depends on a broader community effort beyond the kernel maintainers in the Fedora Engineering team. That needs to precisely not mean more heroics on the part of e.g. QE, rel-eng, etc. I have no idea what the pushback on this issue is, but I'm sure this thread will tell us. But given that Fedora is supposed to encourage such community effort, it would be good to see what people are willing to do to build it.
Making i686 non-release blocking would actually match reality. None of the Fedora Editions appear at all concerned with i686. Cloud is demoting[3] i686 from its offering. Workstation has been fairly ambivalent about it and recommends x86_64. Server does the same. Given the lack of focus on it, and the fact that the broader community is not testing the development releases for i686, I believe this would be a good first step.
"Ambivalent" is probably understated here. It's hard to imagine people securing i686 hardware these days to run a Workstation experience, after all.
https://lists.fedoraproject.org/pipermail/devel/2015-February/208368.html
-- Paul W. Frields http://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ The open source story continues to grow: http://opensource.com --
Perhaps the best approach, from a community perspective, would be to promote a spin to Edition status and recommend *that* for i686 or low resource desktop use cases.
--Pete
On 6 August 2015 at 10:04, Pete Travis lists@petetravis.com wrote:
\
Perhaps the best approach, from a community perspective, would be to promote a spin to Edition status and recommend *that* for i686 or low resource desktop use cases.
--Pete
That would require people volunteering to dedicate time to working on it. Are you volunteering to start this?
On Tue, Aug 04, 2015 at 09:47:27AM -0400, Josh Boyer wrote:
In February[2] we sent out an email highlighting that the kernel team was not going to treat i686 bugs as a priority. Since that time, we have held true to our word and have not focused on fixing i686 bugs at all. It seems that the wider community is also treating i686 similarly. The kernel bug that was made automatic blocker because of existing criteria was present in Fedora since the 4.1-rc6 kernel, which was released May 31. It has been in every boot.iso created since that date. Not a single person reported this issue until last week. That is a timespan of two months.
The kernel team has autotesting for i686 kernels, but the environment there does not utilize boot.iso so it did not detect this. The QA team has automated testing for some of this, but nothing for the i686 architecture at all. It is not a priority there either.
I regularly use i686 and have not done a fresh install since years so would not detect this. Maybe fresh installs aren't such a deal for i686 users and the apparent stability is the reason why it gets less testing. The hardware is not changing so if fresh bugs appear there is a good chance that something else than just i686 is broken?
Appreciate all your efforts and would miss i686. Not a top priority but maybe the memory footprint has some advantages on USB live images?
Richard
On 08/14/2015 12:00 PM, Richard Z wrote:
I regularly use i686 and have not done a fresh install since years so would not detect this. Maybe fresh installs aren't such a deal for i686 users
Well, from my experience, fresh installs on i686 are a major problem w/ Fedora, because Fedora's SW demands have grown, which render it quite likely for i686-users to hit the HW-limitations of their systems, esp. on older i686ers.
and the apparent stability is the reason why it gets less testing.
Agreed.
The hardware is not changing so if fresh bugs appear there is a good chance that something else than just i686 is broken?
Definitely. 10/15 years+ ago, devs were working on 32bit platforms, which had caused packages to have problems on 64bit platforms. Since then, the situation has turned into the converse.
Also, in those days, devs cared about efficiency. Nowadays, they don't care as much, which causes additional problems on i686ers and other low end archs.
Ralf
On Sat, Aug 15, 2015 at 06:47:44AM +0200, Ralf Corsepius wrote:
Definitely. 10/15 years+ ago, [...]
[...]
Also, in those days, devs cared about efficiency. Nowadays, they don't care as much,
People have been making this exact complaint since the 1970s. Probably before.
Am 15.08.2015 um 14:50 schrieb Matthew Miller:
On Sat, Aug 15, 2015 at 06:47:44AM +0200, Ralf Corsepius wrote:
Definitely. 10/15 years+ ago, [...]
[...]
Also, in those days, devs cared about efficiency. Nowadays, they don't care as much,
People have been making this exact complaint since the 1970s. Probably before.
that don't change the fact that 10 years ago Phoenix (the browser which later bacame Firefox), Photoshop, CorelDraw opened at the same time on top of a Windows 2000 Server with a domaincontroller was perfecty running with a Celeron 466 Mhz and 192 MB RAM
so no, there is no excuse that every piece of software these days starts with a ton of ressources
devel@lists.stg.fedoraproject.org
-
Adam Williamson -
Bill Nottingham -
Bruno Wolff III -
Florian Weimer -
Josh Boyer -
Matthew Miller -
Nathanael D. Noblet -
Paul W. Frields -
Pete Travis -
Peter Robinson -
Ralf Corsepius -
Reindl Harald -
Richard Z -
Samuel Sieb -
Stephen John Smoogen