On 7/14/19 1:15 PM, Neal Gompa wrote:

This will also make it impossible for people to locally do multilib build/installs. It will remove COPR’s ability to do the same. For that reason alone, I don’t particularly want this change to happen.

Can you expand on what you mean by 'locally do' ?

Current multilib packages will still be available in the x86_64 repo. Users can still install them from there just fine.

If a user wants to locally build a i686 package, they can use mock against the koji i686 buildroot repo to do so. They could then put that package in a local repo with x86_64 packages and run createrepo on it.

It's true there would be no easily mirrored i386 for them to copy to aviod the internet, but is that really a big use case?

Finally, if you would prefer this not happen now, is there a time when you would further down the road? Whats the critera/goalpost/cutoff?

kevin

On 7/14/19 2:27 PM, Neal Gompa wrote:

Building library packages and making your own multilib repo is impossible without having both the i686 repo and the x86_64 repo, as you need to build for both and then munge them together for a multilib repo.

Sure. Do many people do that anymore?

Historically, we really haven’t wanted people to pull from the Koji repo, and we probably still don’t want to do that, since it’s not mirrored and stressing it could cause more problems our already overtaxed build system environment.

I contend that the number of people who would want to do this is miniscule and in the noise.

No one (sane) would start a 32bit app these days, so likely it's legacy support. The cases I know of are steam and wine. I am sure there's 3rd party items beyond that, but even rhel6 is going to go away soon, so these folks are going to have to come up with some solution really soon.

...

From my point of view, I don’t think it’s worth getting rid of the

32-bit x86 repo until we’re at the point where people would not need to build their own multilib repositories. The cost of generating that for mirroring isn’t that high relative to the amount of pain we’ll cause for external folks trying to build off Fedora.

Fair enough. I see it as already at the time when there are very few of thoese people and we can keep them going for a while by asking them to use the koji buildroot repo.

Think, for example, the repo that shall not be named. That project’s Koji instance pulls in Fedora through the mirrored content as an external source, which feeds its ability to do multilib builds.

Sure, they can repoint that to koji buildroot and keep going.

I’m sorry, but I don’t see us getting rid of this for the foreseeable future without breaking virtually all of our downstreams.

Those few that need to be 32bit applications still. I really don't think this is "all of our downstreams".

kevin

It would be much clearer and user-friendly to move I*86 packages out of the 64 bit repos and make the i*86 an optional add-on

Le July 14, 2019 9:27:03 PM UTC, Neal Gompa ngompa13@gmail.com a écrit :

On Sun, Jul 14, 2019 at 5:21 PM Kevin Fenzi kevin@scrye.com wrote:

...

On 7/14/19 1:15 PM, Neal Gompa wrote:

...

This will also make it impossible for people to locally do multilib build/installs. It will remove COPR’s ability to do the same. For

that

...

...

reason alone, I don’t particularly want this change to happen.

Can you expand on what you mean by 'locally do' ?

Current multilib packages will still be available in the x86_64 repo. Users can still install them from there just fine.

If a user wants to locally build a i686 package, they can use mock against the koji i686 buildroot repo to do so. They could then put

that

...

package in a local repo with x86_64 packages and run createrepo on

it.

...

It's true there would be no easily mirrored i386 for them to copy to aviod the internet, but is that really a big use case?

Finally, if you would prefer this not happen now, is there a time

when

...

you would further down the road? Whats the critera/goalpost/cutoff?

Building library packages and making your own multilib repo is impossible without having both the i686 repo and the x86_64 repo, as you need to build for both and then munge them together for a multilib repo.

Historically, we really haven’t wanted people to pull from the Koji repo, and we probably still don’t want to do that, since it’s not mirrored and stressing it could cause more problems our already overtaxed build system environment.

From my point of view, I don’t think it’s worth getting rid of the 32-bit x86 repo until we’re at the point where people would not need to build their own multilib repositories. The cost of generating that for mirroring isn’t that high relative to the amount of pain we’ll cause for external folks trying to build off Fedora.

Think, for example, the repo that shall not be named. That project’s Koji instance pulls in Fedora through the mirrored content as an external source, which feeds its ability to do multilib builds.

I’m sorry, but I don’t see us getting rid of this for the foreseeable future without breaking virtually all of our downstreams.

-- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Nicolas Mailhot via devel wrote:

It would be much clearer and user-friendly to move I*86 packages out of the 64 bit repos and make the i*86 an optional add-on

+1 to that suggestion.

Vitaly Zaitsev via devel replied:

It will break multilib.

Not really. It would break the legacy "install all packages as multilib if available" mode, which has not been the default modus operandi of YUM for years now. Does DNF even support this mode? (That mode would break because the depsolver would attempt multilibbing all packages including ones which are not intended to be multilib and hence contain multilib conflicts.) But the default "install only the multilibs that are actually required by some 32-bit-only package" mode should work just fine with that setup (and IMHO, that is the only multilib mode that makes sense anyway).

And it would remove the overhead of fetching the metadata for all those multilib packages for all those users running pure 64-bit systems. (It would increase the total metadata for the multilib users though, unless the repository contains only multilib packages, in which case it would be useless to build packages against, including those multilib packages themselves.)

Kevin Kofler

Neal Gompa wrote:

On Mon, Jul 15, 2019 at 7:22 PM Kevin Kofler wrote:

...

Not really. It would break the legacy "install all packages as multilib if available" mode, which has not been the default modus operandi of YUM for years now. Does DNF even support this mode? (That mode would break because the depsolver would attempt multilibbing all packages including ones which are not intended to be multilib and hence contain multilib conflicts.) But the default "install only the multilibs that are actually required by some 32-bit-only package" mode should work just fine with that setup (and IMHO, that is the only multilib mode that makes sense anyway).

This is definitely supported, as this is how multilib works on Mageia and OpenMandriva today, both of which use DNF too.

But that doesn't mean Fedora has to continue supporting the broken mode. ;-) Why would you want to install all libraries as multilib even if nothing 32- bit uses them? It's just a huge waste of bandwidth and disk space.

Kevin Kofler

On Sun, 14 Jul 2019 at 18:16, Neal Gompa ngompa13@gmail.com wrote:

On Sun, Jul 14, 2019 at 5:21 PM Kevin Fenzi kevin@scrye.com wrote:

...

On 7/14/19 1:15 PM, Neal Gompa wrote:

...

This will also make it impossible for people to locally do multilib build/installs. It will remove COPR’s ability to do the same. For that reason alone, I don’t particularly want this change to happen.

Can you expand on what you mean by 'locally do' ?

Current multilib packages will still be available in the x86_64 repo. Users can still install them from there just fine.

If a user wants to locally build a i686 package, they can use mock against the koji i686 buildroot repo to do so. They could then put that package in a local repo with x86_64 packages and run createrepo on it.

It's true there would be no easily mirrored i386 for them to copy to aviod the internet, but is that really a big use case?

Finally, if you would prefer this not happen now, is there a time when you would further down the road? Whats the critera/goalpost/cutoff?

Building library packages and making your own multilib repo is impossible without having both the i686 repo and the x86_64 repo, as you need to build for both and then munge them together for a multilib repo.

Historically, we really haven’t wanted people to pull from the Koji repo, and we probably still don’t want to do that, since it’s not mirrored and stressing it could cause more problems our already overtaxed build system environment.

From my point of view, I don’t think it’s worth getting rid of the 32-bit x86 repo until we’re at the point where people would not need to build their own multilib repositories. The cost of generating that for mirroring isn’t that high relative to the amount of pain we’ll cause for external folks trying to build off Fedora.

Think, for example, the repo that shall not be named. That project’s Koji instance pulls in Fedora through the mirrored content as an external source, which feeds its ability to do multilib builds.

I’m sorry, but I don’t see us getting rid of this for the foreseeable future without breaking virtually all of our downstreams.

The problem is: 1. We currently have no idea of who these downstreams are 2. We have no idea of what they want? [Do they need every package.. do they need just N packages?]

The turn around of this argument is that we can never allow ANY change/update in our infrastructure or to any package because it would break some downstream sometime. And yet we do allow updates to gcc/kernel as it is built into what we do.. aka we have decided already there is a cut-off where people are going to be 'broken/left behind' for some reason all the time. The expectation is that sometimes we will lose them permanently but a lot of time they will catch up or get ahead of us.

We do stop some changes because we have a way to get a feeling that the number of people affected by the change are too many to deal with. But we have to have a way to measure what that is.. even if it is a back of the envelope. How can we get an idea on these?

Because if we are going to start down the 'we can't change something because it could break some unknown sized group' ... it will quickly morph into Fedora 31 being our last release because we have enough rules lawyers in Fedora to keep any and every update from happening because it could break someone. 'Sorry you can't push that CVE, it breaks my hacking group which uses these to break into systems.' will become the releng or fesco ticket of the hour.

That said, I think the problem with this change is that everyone's mental image of this change is different. From various other threads.. people have different ideas of what multi-lib does, what it provides, who uses it, how does it get built, what is required to build it, and what does an x86_64 user get of i686 packages.

On 7/14/19 2:35 PM, John Reiser wrote:

Kevin Fenzi wrote:

...

Neal Gompa wrote:

[[snip]]

...

...

This will also make it impossible for people to locally do multilib build/installs. It will remove COPR’s ability to do the same. For that reason alone, I don’t particularly want this change to happen.

...

Can you expand on what you mean by 'locally do' ?

I want to run "gcc -m32 -o my_app-i686 *.o ..." locally on my own box to build executables that run as 32-bit apps on multilib x86_64. For some apps 2GB of malloc() arena is plenty, and they run faster in 32-bit mode because a 64-byte cache line contains 16 pointers instead of only 8.

This should still work, unless there are libraries you use that are not multilib.

[[snip]]

...

Finally, if you would prefer this not happen now, is there a time when you would further down the road? Whats the critera/goalpost/cutoff?

One year after Red Hat Enterprise Linux version 7 reaches end-of-support. It would be handy for Fedora to have 32-bit *-devel packages until then.

We will still have 32bit devel packages in the x86_64 repos after this change. It doesn't affect multilib at all. It only stops building and publishing the 'pure' i386 repos on the mirror network.

I don't think we can drop multilib until at least steam/wine are ready for it at least.

kevin

Hello, Dridi Boukelmoune.

Mon, 15 Jul 2019 06:59:33 +0000 you wrote:

game that cannot move to 64bit support because it's dragging binaries for which it doesn't have source code.

Wine64 can still emulate 32-bit WinPE executables.

-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org)

Hello, Jiri Vanek.

Mon, 15 Jul 2019 09:22:57 +0200 you wrote:

That is not enough. See what hapened to Ubuntu once they dropped i686

They decided to remove whole 32-bit support, including multilib support.

We need to drop 32-bit packages, except needed to run Steam and Wine32.

Third-party developers like Valve should start building native 64-bit executables and stop using vulnerable ancient architectures.

-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org)

On Mon, Jul 15, 2019 at 7:58 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:

Hello, Dridi Boukelmoune.

Mon, 15 Jul 2019 06:59:33 +0000 you wrote:

...

game that cannot move to 64bit support because it's dragging binaries for which it doesn't have source code.

Wine64 can still emulate 32-bit WinPE executables.

Emulate as in not run natively even though the hardware might be able to?

Thanks for the tip anyway, it might be worth giving a try.

Dridi

On 7/15/19 11:34 AM, Vitaly Zaitsev via devel wrote:

Hello, Dridi Boukelmoune.

Mon, 15 Jul 2019 08:26:59 +0000 you wrote:

...

Emulate as in not run natively even though the hardware might be able to?

Sorry for misinformation. Wine64 is still require 32-bit libraries in order to run legacy 32-bit Windows PE executables.

https://wiki.winehq.org/FAQ#Is_there_a_64_bit_Wine.3F

Exactly. I really do not understand this rush change. The original change, to remove only 32 kernel is definitely the way to go. But This one, should be postponded to f32, with much better contingency plan.

-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

On Mon, Jul 15, 2019 at 12:45 PM Jiri Vanek jvanek@redhat.com wrote:

On 7/15/19 11:34 AM, Vitaly Zaitsev via devel wrote:

...

Hello, Dridi Boukelmoune.

Mon, 15 Jul 2019 08:26:59 +0000 you wrote:

...

Emulate as in not run natively even though the hardware might be able to?

Sorry for misinformation. Wine64 is still require 32-bit libraries in order to run legacy 32-bit Windows PE executables.

https://wiki.winehq.org/FAQ#Is_there_a_64_bit_Wine.3F

Exactly. I really do not understand this rush change. The original change, to remove only 32 kernel is definitely the way to go. But This one, should be postponded to f32, with much better contingency plan.

I agree. No longer building kernels and composing bootable images for i686 is a relatively small change with limited user impact. But stopping to compose i686 repositories at all, which means adapting multiple processes (mock, copr, etc.), is throwing out the baby with the bathwater.

(I for one am still semi-regularly building stuff in the mock fedora-rawhide-i686 chroots, especially for testing if things still work on 32bit systems (be it armv7hl or i686) - I know that this will probably continue to work somehow, but making it more complicated is hostile to developers and packagers who care about their packages working on all architectures.)

Fabio

...

-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

-- Jiri Vanek Senior QE engineer, OpenJDK QE lead, Mgr. Red Hat Czech jvanek@redhat.com M: +420775390109 _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

On Mon, Jul 15, 2019, 19:33 Kevin Fenzi kevin@scrye.com wrote:

On 7/15/19 5:43 AM, Fabio Valentini wrote:

...

(I for one am still semi-regularly building stuff in the mock fedora-rawhide-i686 chroots, especially for testing if things still work on 32bit systems (be it armv7hl or i686) - I know that this will probably continue to work somehow, but making it more complicated is hostile to developers and packagers who care about their packages working on all architectures.)

Well, it wouldn't be more complicated. Mock would just download from a different repo in that case.

Also, if it's 32bit you are testing, perhaps some cheap armv7 device would be a better platform moving forward?

kevin

I have a Raspberry Pi 2 Model B somewhere, but the experience of trying to build some packages on it a while ago was ... testing my patience, let's say. Maybe a faster SD card would help, but I'm guessing that using "--forcearch armv7hl" to force qemu emulation in mock on my main PC's pretty beefy CPU would still be a better experience. Additionally there's no 32bit arm chroots supported in COPR, which complicates the situation for 32bit support even more.

But that's a bit beside the point I wanted to make. I've been struggling to keep the Java stack building at all (mostly Stewardship SIG packages), and the removal of 32bit support in eclipse has made that even more difficult.

Even still, dozens of Java packages either won't build or won't install on 32bit arches because of that (mostly because gradle isn't installable anymore, because jgit is gone on 32bit). Even the modular Java packages don't support 32bit arches anymore.

If we start to relegate i686 to a tertiary architecture only intended for multilib use (which is basically the effect of this proposal), there'll be even fewer reasons to fix things or keep things working on 32bit. I mean, I'm already tempted to just start dropping things ... because, well, it's not like anybody seems to care about those packages anyway.

Fabio

---

devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

On Mon, Jul 15, 2019 at 2:53 PM Fabio Valentini decathorpe@gmail.com wrote:

On Mon, Jul 15, 2019, 19:33 Kevin Fenzi kevin@scrye.com wrote:

...

On 7/15/19 5:43 AM, Fabio Valentini wrote:

...

(I for one am still semi-regularly building stuff in the mock fedora-rawhide-i686 chroots, especially for testing if things still work on 32bit systems (be it armv7hl or i686) - I know that this will probably continue to work somehow, but making it more complicated is hostile to developers and packagers who care about their packages working on all architectures.)

Well, it wouldn't be more complicated. Mock would just download from a different repo in that case.

Also, if it's 32bit you are testing, perhaps some cheap armv7 device would be a better platform moving forward?

kevin

I have a Raspberry Pi 2 Model B somewhere, but the experience of trying to build some packages on it a while ago was ... testing my patience, let's say. Maybe a faster SD card would help, but I'm guessing that using "--forcearch armv7hl" to force qemu emulation in mock on my main PC's pretty beefy CPU would still be a better experience. Additionally there's no 32bit arm chroots supported in COPR, which complicates the situation for 32bit support even more.

But that's a bit beside the point I wanted to make. I've been struggling to keep the Java stack building at all (mostly Stewardship SIG packages), and the removal of 32bit support in eclipse has made that even more difficult.

Even still, dozens of Java packages either won't build or won't install on 32bit arches because of that (mostly because gradle isn't installable anymore, because jgit is gone on 32bit). Even the modular Java packages don't support 32bit arches anymore.

If we start to relegate i686 to a tertiary architecture only intended for multilib use (which is basically the effect of this proposal), there'll be even fewer reasons to fix things or keep things working on 32bit. I mean, I'm already tempted to just start dropping things ... because, well, it's not like anybody seems to care about those packages anyway.

And honestly, i686 is our only useful 32-bit architecture from a development standpoint. People can iterate quickly on it, and easily test it.

It is incredibly difficult to deal with 32-bit only issues if armv7hl is the only thing we have.

On 7/15/19 1:51 PM, Stephen John Smoogen wrote:

OK could all the people who are so interested in i686 get onto the x86_32 mailing list and chime up there about what they are wanting to do? Also start having regular sig meetings and other things which have been pretty dead for about a year?

My guess is that most users are those using wine or steam and those users are unlikely to see this. I've heard of a few who still have 32-bit only computers, but those are getting fewer. (e.g. I'm no longer one of those.)

I totally agree with that view. Making such decisions without public discussion is not respecting user's freedom of choice. And this list doesn't count as a public discussion. Nobody will know about it outside a very closed circle. If you don't know exact numbers or reasons why people still use that architecture, then rushing to drastic measures just won't have enough rationale and will be viewed as a lack of care.

The reasons to use 32-bit userland might be more than you think. There are different use scenarios besides browsing Internet or virtualization. There could be a high cost of upgrade (think about replacing many computers at once). There could be a unique peripheral equipment. There could be compatibility concerns. And there could be just not enough memory.

OTOH, what could justify removing repository entirely? Why not make it an option which could be installed on demand? That would be better than leave everyone out on the cold. Is the cost to build it really that high? Safety could not be a reason in a tightly contained environment. Let users decide for themselves.

There is no black and white distinction: multilib or i686 kernel. There are reasons why using x86_64 kernel with i686 userland might be a better option. Some older office computers were manufactured with 64-bit CPU but without support for more than 4 GB RAM. In such case using i686 userland on a x86_64 kernel provides much more free memory than using 64-bit userland. Such configuration is unsupported, but neither is i686. Giving users a choice is always better than decide for them behind their backs.

And of course there is still an option to switch to another OS. Do I need to remind that Linux and Red Hat were not created just to replace some other OS, but to respect freedom of choice? What happened that this is not even mentioned in such discussions? Is this just business as usual?

That's nice to know Fedora's developers point of view on that subject. But I'm not subscribing to that view. I'm with Richard Stallman. And now I clearly see why he is opposed to OSS paradigm. Looks like I was in a wrong place for all these years. Time to move elsewhere.

Boy, am I glad you've said that. I was waiting for it.

But looks like you are mistaken. First of all, it's not one, but at least two of them. Second, nobody else seems to be supporting your point.

May be there are more interested people that we know, but they are not reading that list. There will just be just every man for himself and Fedora has failed to recognize that.

This requires time and effort too. Nobody will appear just by a miracle. I recognize that there is much less people interested in this architecture but it's much more than zero.

There is no either right or wrong stance here. We are discussing possible alternatives to "just drop it" attitude.

What work should be done? Please, be more specific. Right now I'm running a i686 userland and it works. If I would be able to build the whole repository myself I'm pretty sure that most things will still work. If it won't work I might try to fix it and contribute patches back. But without that repository I can't even try it in the first place.

You are just pushing me and others away, so we should go use other distributions which provide ready to run builds. And I'm not talking about i686 *kernel* anywhere. We are talking about *userland* only. I'm running 64-bit CPU all along, but I have limited memory. Others could use laptops with restricted memory which would be a performance hit if they start using x86_64 userland.

You are not providing any alternative but starting to build everything ourselves or stop using Fedora and move elsewhere.

First of all thanks for the link. It just proves that the SIG's expectations were too high.

If I understand it all correctly, the main reason to drop i686 repo was the mailing list inactivity? Is that right? So everyone interested in that architecture is now deprived from using it on Fedora because some formalities were not met? And if I have no time to participate in that list, I can't fix problems myself? Because without that repository I'm forced to use other distribution.

That's just... weird.

But how do you now that I'm not fixed it myself and forgot to post on that list? Or that I'm even just used to live with that bug and just don't want to spend all my time chasing it?

I'm pretty sure that I can point point out bugs in official Fedora repository that were dormant for several years without any conclusion and nobody dropped support for all those applications just for that reason.

Anyway, I'm not expecting that something will change because of that discussion. It is just bad that the interests of users are of a lower priority then some purely bureaucratic reasons.

I would argue that it might be difficult to distinguish work needed to find out if it was i686 specific when there already is similar bug on x86_64. Also, it's difficult to rate bug importance for most users. As I've already said that I was completely satisfied with the status quo and it was a big surprise for me to discover that I just won't be able to upgrade to the next version. Also, this discovery was purely accidental because there is no announcements anywhere I could see.

Anyway, I would be prepared to fix things myself if such possibility was given to me. But alas there is no choice now.

Even better. That means that you can still get support for x86 but it will require some more work on the user's side. They should just check if that bug is indeed i686 specific.

I believe that all that argument for the lats three days was completely unnecessary and should be blamed on an utterl failure of communication.

Yes, that's understandable. But this is beating of a dead horse.

But what matters now is that by doing some small investigation i686 users can still get support for their bugs which are common for both platforms. This doesn't require any formalities like SIG or commitments which they can't make and it is always available for anyone who can afford to spend some additional time if such bug affects them bad enough.

I think this could work better than previous attempts at keeping x86 SIG alive. Of course nothing prevents some volunteers to do above work on behalf of other users or create mirrors for distribution of i686 packages. But this is not critical to keep things running.

But there should be some reason for that lack of interested volunteers in Fedora. Right now I'm looking at stats for other distributions which are not going to drop i686 any time soon, e.g. Debian, NixOS, Gentoo. There must me some very fundamental difference with how they operate. Of course one of the reasons might be that some are relying on users to build packages themselves. OTOH they have their own CI, binary repositories/caches and Debian only has binaries for its package management. Why the difference? I'm not sure...

Maybe in other distros, people interested in i686 support actually do something about it instead of talking and talking and talking about it on mailing lists?

Maybe someone with so much free time on their hands could maintain such a kernel in Fedora by applying downstream packages of such a distro.

That person'd need to find a distribution that goes at least as fast Fedora when it comes to upgrading kernel packages... I very much doubt we'll find one we could rely on.

Dridi

Hello Nicolas,

On Fri, Sep 13, 2019 at 10:51 AM Nicolas Chauvet kwizart@gmail.com wrote: <snip>

Well... I don't qualify as a person with much free time but...

I'm toying with kernel-longterm in a copr for .4.19 branch, and I've enabled i686 there. The rebuilt is a semi-automated way. This i686 build is totally untested, please send patch along to report any issue (or report upstream if relevant). https://copr.fedorainfracloud.org/coprs/kwizart/kernel-longterm-4.19/

While I appreciate the effort, this also doesn't work in the long term (pun intended) once we start packaging or upgrading software that relies on whatever Fedora's kernel provides that isn't in 4.19 we break that piece of software...

Dridi

On Mon, Sep 09, 2019 at 06:23:18PM -0000, vvs vvs wrote:

But how do you now that I'm not fixed it myself and forgot to post on that list? Or that I'm even just used to live with that bug and just don't want to spend all my time chasing it?

It's simple; if you (and everyone else) doesn't say anything in the public discusison channels, or generate _some_ sort of activity in other project channels (eg bugzilla, irc, whatever) then it is a perfectly reasonable assumption to make that you are not doing anything of general relevance to Fedora.

I'm pretty sure that I can point point out bugs in official Fedora repository that were dormant for several years without any conclusion and nobody dropped support for all those applications just for that reason.

Every Fedora release has packages dropped due to failures to compile or other problems that run afoul of packaging policies, with nobody stepping up to fix them.

Anyway, I'm not expecting that something will change because of that discussion. It is just bad that the interests of users are of a lower priority then some purely bureaucratic reasons.

I'm sorry you feel that way.

- Solomon

On Mon, Sep 09, 2019 at 07:09:49PM -0000, vvs vvs wrote:

Ok, now I see that Fedora is just for activists. If I'm not one of them then I don't deserve any possibility to use it and should blame myself. Thanks for explaining it to me.

If I may quote from the landing page on https://getfedora.org/

"Fedora creates an innovative, free, and open source platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users."

...

"Fedora Workstation is a polished, easy to use operating system for laptop and desktop computers, with a complete set of tools for developers and makers of all kinds."

Using Fedora (or not) has always been your choice.

Good day,

- Solomon

Well, thanks for sharing.

I'm not complaining that nobody wants to fix things for me. I'm complaining because there is no possibility to fix things myself. After removing i686 repository I'm either should start building it myself or switch to another distribution. I'm not trying to hurt anyone's feelings, I just have no other choice. If there is no possibility to keep that repository than it's fine, but I was not convinced that there are other reasons for that decision aside bureaucratic ones and lack of empathy. If putting that repository on some optional host for anyone to be able to fix it themselves would severely harm the project then I was wrong all along and I'm really sorry.

And I thought that should be obvious, silly me. Just kidding.

Of course I would do it if there were no better choice. I'm just struggling to find out if there is no other possibility whatsoever. There might be reasons why Fedora is just unable to keep it updated that I don't know. And of course I could use another repository provided by some other distribution. I'm just trying to find out what are my options. I would prefer to keep using modern Fedora unless I'm forced not to.

Just in case. There is no reason to believe that I'm upset or frustrated. That's just a conversation which might get heated at a time for various reasons which are not directly related to the subject.

On Monday, September 9, 2019 12:44:42 PM MST DJ Delorie wrote:

"vvs vvs" vvs009@gmail.com writes:

...

Ok, now I see that Fedora is just for activists. If I'm not one of them then I don't deserve any possibility to use it and should blame myself. Thanks for explaining it to me.

I think you're overreacting a bit, but there is some truth in this. Fedora is created and maintained by the community. You are part of the community. If enough of the community shares your needs, some fraction of those will step up to do the work, and you all benefit. If your needs aren't shared by enough of the community, either you need to do it all yourself (or pay someone to act on your behalf), or your needs will never get met.

This has nothing to do with "deserve" or "blame" - it's just numbers. Most people have switched to 64-bit, so most work is done for 64-bit, even if not all the 64-bit users are also contributors.

The 32-bit community has shrunk to the point where there aren't enough contributors to keep the builds building and the fixes fixing, and there are real problems backing up because of that, even if they don't affect you personally. When there are enough problems and no contributors, what other choice do we have? It's broken and nobody is fixing it.

Thus comes the hard part of any project - put up or shut up. Harsh, but it's the root of how things get done - they get done by people doing them. Do or do not, there is no sit-on-the-mailing-list-and-hope.

Back when I started the DJGPP project, I had to do everything myself. The community grew and there were lots of contributors. Then the community shrunk until we're back down to 2 people doing all the work. Thus is the cycle of projects, but I don't complain that not enough people are still using DOS :-)

OTOH you won't hurt our feelings if you switch distros. Go where your community is ;-)

While most users on Intel/AMD based systems are now running x64 kernels, most proprietary software released for various GNU/Linux distros are 32 bit.

On Mon 9. Sep 2019 at 21:45, DJ Delorie dj@redhat.com wrote:

"vvs vvs" vvs009@gmail.com writes:

...

Ok, now I see that Fedora is just for activists. If I'm not one ofn them then I don't deserve any possibility to use it and should blame myself. Thanks for explaining it to me.

I think you're overreacting a bit, but there is some truth in this. Fedora is created and maintained by the community. You are part of the community. If enough of the community shares your needs, some fraction of those will step up to do the work, and you all benefit. If your needs aren't shared by enough of the community, either you need to do it all yourself (or pay someone to act on your behalf), or your needs will never get met.

This has nothing to do with "deserve" or "blame" - it's just numbers. Most people have switched to 64-bit, so most work is done for 64-bit, even if not all the 64-bit users are also contributors.

The 32-bit community has shrunk to the point where there aren't enough contributors to keep the builds building and the fixes fixing, and there are real problems backing up because of that, even if they don't affect you personally. When there are enough problems and no contributors, what other choice do we have? It's broken and nobody is fixing it.

Thus comes the hard part of any project - put up or shut up. Harsh, but it's the root of how things get done - they get done by people doing them. Do or do not, there is no sit-on-the-mailing-list-and-hope.

Back when I started the DJGPP project, I had to do everything myself. The community grew and there were lots of contributors. Then the community shrunk until we're back down to 2 people doing all the work. Thus is the cycle of projects, but I don't complain that not enough people are still using DOS :-)

OTOH you won't hurt our feelings if you switch distros. Go where your community is ;-) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

On Monday, September 9, 2019 12:09:49 PM MST vvs vvs wrote:

Ok, now I see that Fedora is just for activists. If I'm not one of them then I don't deserve any possibility to use it and should blame myself. Thanks for explaining it to me.

Please don't let the hostilities of this list get to you, there are those of us in Fedora that want to help users like you. I'm one of them, and I'd be more than happy to step up to the plate and get to work on "supporting" x86 based systems. I've got systems shipping out to me now from my old location.

* vvs vvs [10/09/2019 11:41] :

And the primary reason why that SIG initiative never worked is that the project didn't put any significant efforts to make that happen.

I'm going to disagree with you here.

Back in 2017, we went through a very long discussion which lead to the creation of the x86 SIG. The discussion on the mailing list started off at a reasonable rate and then slowed to a stop because it couldn't keep its momentum going.

So the fault with the SIG not working really lies with the people who wanted to keep x86_32 going not being able/willing to do the work necessary, not all the other Fedora developers.

Either we can be a low activity grass-roots enthusiasts and find our own ways to contribute packages, e.g. COPR might be useful.

This is pretty much the definition of a SIG.

Emmanuel

* vvs vvs [10/09/2019 21:07] :

I suppose that SIG is a much formal entity than just a bunch of individuals performing some non-regular activities.

The PHP SIG is barely more than one person, the Perl SIG has no regular meeting and the mailing list activity is not representative of the work being done but thoses SIGS still do the work necessary to keep their stack in the distribution.

If that really was that simple then just subscribing to that list would be enough to get support for x86 architecture and we wouldn't be here.

I think you're confusing cause and effect, here.

We look at Bugzilla stats and mailing-lists activity because it's (supposed to be) indicative of the work being done. It's a by-product, not the final outcome. Subscribing to a mailing-list (and even posting to it) isn't going to improve the packages in the distribution by itself.

Emmanuel

On 9/10/19 4:28 PM, vvs vvs wrote:

But that's actually the same that I was trying to say. Meeting that activity statistics is the essence of such formal group. But grass-roots enthusiasts don't have such commitments. They can do some work occasionally if time allows but there is no strict agenda. This contradicts those expectations which you describe. So while there are people ready to do some work sometimes they just don't meet those criteria and this is not enough to be able to call them SIG.

And that's the whole problem here. Those people are not able to do enough work to keep i686 going. As has been said here many times, if you can get enough people to do the necessary work, then there is no need for the i686 repos to go away. But that just hasn't happened.

I don't think there are any requirement to create a SIG other than to have a group of people interested in the topic. The SIG still exists, but it isn't doing enough to be able to keep i686 alive.

And even that might not be necessary at all because most bugs are common between 32 and 64-bit. Honestly, I don't think such SIG was really needed after all.

On Monday, September 9, 2019 8:36:45 AM MST vvs vvs wrote:

There is no either right or wrong stance here. We are discussing possible alternatives to "just drop it" attitude.

What work should be done? Please, be more specific. Right now I'm running a i686 userland and it works. If I would be able to build the whole repository myself I'm pretty sure that most things will still work. If it won't work I might try to fix it and contribute patches back. But without that repository I can't even try it in the first place.

You are just pushing me and others away, so we should go use other distributions which provide ready to run builds. And I'm not talking about i686 *kernel* anywhere. We are talking about *userland* only. I'm running 64-bit CPU all along, but I have limited memory. Others could use laptops with restricted memory which would be a performance hit if they start using x86_64 userland.

You are not providing any alternative but starting to build everything ourselves or stop using Fedora and move elsewhere.

There's no reason to drop x86 kernel builds either.

On Tuesday, September 10, 2019 9:54:50 AM MST Kevin Fenzi wrote:

On 9/9/19 9:34 PM, John M. Harris Jr. wrote:

...

There's no reason to drop x86 kernel builds either.

Sure there are... from the change page:

"The i686 kernel is of limited use as most x86 hardware supports 64bit these days. It has been in a status of "community supported" for several Fedora releases now. As such, it gets very little testing, and issues frequently appear upstream. These tend to go unnoticed for long periods of time. When issues are found, it is often a long time before they are fixed because they are considered low priority by most developers upstream. This can leave other architectures waiting for important updates, and provides a less than desirable experience for people choosing to run a 32bit kernel. With this proposal, the i686 kernel will no longer be built. A kernel headers package will still exist, and all 32bit packages should continue to build as normal. The main difference is there would no longer be bootable 32bit images."

The lack of fixes for i686 kernels is slowing down all the other arches that are supported (and thus all of fedora).

kevin

The first sentence of that paragraph is simply incorrect, new hardware doesn't change what old hardware supports, nor does the availability of new hardware replace old hardware in itself.

- - John Harris

On Wednesday, September 11, 2019 12:28:31 AM MST Samuel Sieb wrote:

On 9/10/19 11:01 PM, John M. Harris Jr. wrote:

...

On Tuesday, September 10, 2019 9:54:50 AM MST Kevin Fenzi wrote:

...

Sure there are... from the change page:

"The i686 kernel is of limited use as most x86 hardware supports 64bit these days. It has been in a status of "community supported" for several

[snip]

...

...

The lack of fixes for i686 kernels is slowing down all the other arches that are supported (and thus all of fedora).

The first sentence of that paragraph is simply incorrect, new hardware doesn't change what old hardware supports, nor does the availability of new hardware replace old hardware in itself.

It's not incorrect. Almost all x86 hardware is 64-bit capable, therefore building a 32-bit is of very limited use. It is not easy to find 32-bit only CPUs now. Yes, I know some still exist; I have one embedded in my wall (NSC Geode). But the last paragraph is important. Keeping the i686 kernel in Fedora is hurting everyone for the benefit of an extremely small group of users.

Again, it's completely false that "almost all x86 hardware is 64-bit capable". That may be true of newer hardware, but that does nothing to change existing hardware. The laptop I'm typing this email on right now is 32 bit only, by the way. It was manufactured in 2011.

I also fail to see how keeping the i686 kernel "is slowing down all other arches that are supported", and I'd love to know why that's the case, so I can get to fixing it.

- - John Harris

On Wed, Sep 11, 2019 at 2:51 AM John M. Harris Jr. johnmh@splentity.com wrote:

On Wednesday, September 11, 2019 12:28:31 AM MST Samuel Sieb wrote:

...

On 9/10/19 11:01 PM, John M. Harris Jr. wrote:

...

On Tuesday, September 10, 2019 9:54:50 AM MST Kevin Fenzi wrote:

...

Sure there are... from the change page:

"The i686 kernel is of limited use as most x86 hardware supports 64bit these days. It has been in a status of "community supported" for several

[snip]

...

...

The lack of fixes for i686 kernels is slowing down all the other arches that are supported (and thus all of fedora).

The first sentence of that paragraph is simply incorrect, new hardware doesn't change what old hardware supports, nor does the availability of new hardware replace old hardware in itself.

It's not incorrect. Almost all x86 hardware is 64-bit capable, therefore building a 32-bit is of very limited use. It is not easy to find 32-bit only CPUs now. Yes, I know some still exist; I have one embedded in my wall (NSC Geode). But the last paragraph is important. Keeping the i686 kernel in Fedora is hurting everyone for the benefit of an extremely small group of users.

Again, it's completely false that "almost all x86 hardware is 64-bit capable". That may be true of newer hardware, but that does nothing to change existing hardware. The laptop I'm typing this email on right now is 32 bit only, by the way. It was manufactured in 2011.

I also fail to see how keeping the i686 kernel "is slowing down all other arches that are supported", and I'd love to know why that's the case, so I can get to fixing it.

People offered to "get to fixing it" 2 years ago when this was first proposed. They organized, started a SIG, and then crickets. Turns out the fixing it takes continuous effort. This isn't a one time thing, it is issues that pop up regularly.

Yes, thanks. Sadly, I see that I have no choice but to switch to another distribution even though I'm using 64-bit CPU. It's just that the memory can't be upgraded and buying new computer just to keep running Fedora is not viable. It's 12 years old, is in good condition and I'm completely satisfied with its performance for my needs. I wonder what owners of thin terminals will do if they used Fedora, especially if there are many of them. The cost of upgrading some old terminals for some school can be too high.

Maintaining my own distribution is a little too much for me at the moment.

On Mon, 2019-09-09 at 13:27 -0500, Bruno Wolff III wrote:

On Mon, Sep 09, 2019 at 18:06:02 -0000, vvs vvs vvs009@gmail.com wrote:

...

Yes, thanks. Sadly, I see that I have no choice but to switch to another distribution even though I'm using 64-bit CPU. It's just that the memory can't be upgraded and buying new computer just to keep running Fedora is not viable. It's 12 years old, is in good condition and I'm completely satisfied with its performance for my needs. I wonder what owners of thin terminals will do if they used Fedora, especially if there are many of them. The cost of upgrading some old terminals for some school can be too high.

It is probably very rare for someone to have just enough memory for a system to run reasonably using i686, but tank when using x86_64. If there is some key code that causes the problem, you might be able to rebuild that code to use 32 bit addresses and save enough memory to make things work reasonably.

Yeah, I've recently switched an old Atom A330[0] based system[1] with 2 GB of RAM (that's the maximum it supports) from a 32-bit to a 64-bit based distro (after finding out it can actually run 64-bit code). It has been running just fine and actually feels a bit faster now.

[0] https://ark.intel.com/content/www/us/en/ark/products/35641/intel-atom-proces... [1] https://ark.intel.com/content/www/us/en/ark/products/42491/intel-desktop-boa...

---

devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org based

On 9/9/19 11:47 AM, Martin Kolman wrote:

Yeah, I've recently switched an old Atom A330[0] based system[1] with 2 GB of RAM (that's the maximum it supports) from a 32-bit to a 64-bit based distro (after finding out it can actually run 64-bit code). It has been running just fine and actually feels a bit faster now.

I had a similar experience. I setup a bunch of old P4 desktops for a school computer lab. Initially, I used a 32-bit install, but then I discovered that they actually supported 64-bit. A Mate desktop runs just fine in 2GB of RAM even with Firefox and Libreoffice.

On Mon, Sep 09, 2019 at 07:01:59PM -0000, vvs vvs wrote:

No, I don't think so. I'm using some (non Fedora related) applications which use every bit of available memory. It's a bit stressed just as it is, but losing additional couple of megabytes for no useful reason will be too much a hit. And I can't change their code, because that codebase is big and complex (as usual) and I just don't have enough time to do everything myself.

Honestly, it sounds like your 12-year-old system is barely adequate for your needs, and even a minimally-newer system capable of holding more RAM would pay for itself pretty rapidly.

(Unless you don't value your own time or stress levels..)

- Solomon

On Mon, Sep 09, 2019 at 19:01:59 -0000, vvs vvs vvs009@gmail.com wrote:

No, I don't think so. I'm using some (non Fedora related) applications which use every bit of available memory. It's a bit stressed just as it is, but losing additional couple of megabytes for no useful reason will be too much a hit. And I can't change their code, because that codebase is big and complex (as usual) and I just don't have enough time to do everything myself.

Have you actually tested this? That is very odd behavior.

On Monday, September 9, 2019 10:29:23 AM MST Bruno Wolff III wrote:

On Mon, Sep 09, 2019 at 14:52:07 -0000, vvs vvs vvs009@gmail.com wrote:

...

May be there are more interested people that we know, but they are not reading that list. There will just be just every man for himself and Fedora has failed to recognize that.

...

This requires time and effort too. Nobody will appear just by a miracle. I recognize that there is much less people interested in this architecture but it's much more than zero.

I'm probably one of the few people still running Fedora on a machine that uses i686, that can't use x86_64. The machine is around 15 years old and is costly to get replacement parts for and I'm running out of spares. I was supposed to replace the machine last month, but needed another month to save up enough to buy the rest of the replacement. I've actually work with upstream to get kernel bugs fixed for this machine.

Unfortunately I run rawhide and things got shut down a little sooner than I hoped, so I'm not getting updates right now and don't want to go back to f30 with the short horizon for retirement (though I did grab an f30 kernel).

I don't think you are going to find many people who both run Fedora and have to use i686.

There is a cost to keeping things running on i686 and it doesn't look like it is worth paying right now. And things are looking to get worse rather than better.

You have options. You can switch to another distro that will support i686 for a while yet. Use f30 until it's EOL (or beyond if the machines are isolated). Or maintain your own distro. The tools for Fedora are open, so you could set up your own koji instance drawing from Fedora and applying your fixes where needed. Getting started will probably be hard, but once things are running you'll be OK until there is a key bug you can't get fixed.

There are at least 4 people in this thread alone that are running Fedora on x86 systems.

On Tue, 10 Sep 2019 08:00:52 -0400 Neal Gompa ngompa13@gmail.com wrote:

On Tue, Sep 10, 2019 at 12:24 AM John M. Harris Jr. johnmh@splentity.com wrote:

...

On Monday, September 9, 2019 10:29:23 AM MST Bruno Wolff III wrote:

...

On Mon, Sep 09, 2019 at 14:52:07 -0000, vvs vvs vvs009@gmail.com wrote:

[...]

...

[...]

...

...

I'm probably one of the few people still running Fedora on a machine that uses i686, that can't use x86_64. The machine is around 15 years old and is costly to get replacement parts for and I'm running out of spares. I was supposed to replace the machine last month, but needed another month to save up enough to buy the rest of the replacement. I've actually work with upstream to get kernel bugs fixed for this machine.

Unfortunately I run rawhide and things got shut down a little sooner than I hoped, so I'm not getting updates right now and don't want to go back to f30 with the short horizon for retirement (though I did grab an f30 kernel).

I don't think you are going to find many people who both run Fedora and have to use i686.

There is a cost to keeping things running on i686 and it doesn't look like it is worth paying right now. And things are looking to get worse rather than better.

You have options. You can switch to another distro that will support i686 for a while yet. Use f30 until it's EOL (or beyond if the machines are isolated). Or maintain your own distro. The tools for Fedora are open, so you could set up your own koji instance drawing from Fedora and applying your fixes where needed. Getting started will probably be hard, but once things are running you'll be OK until there is a key bug you can't get fixed.

There are at least 4 people in this thread alone that are running Fedora on x86 systems.

Do these 4 people want to revive the x86 SIG and take on the work of dealing with 32-bit x86 bugs? Because they *do* exist and do cause problems...

I'm also using several Fedora/i686 machines and I'd like to help keep this branch. As often, the problem is with the lack of time and the fact that I do not know the full range of work to be done. But if one of the current experienced i686 maintainers is willing to continue, I'll join. I have some experience with creating and managing RPM packages. Can I register somewhere as interested in this job?

I'm sorry, but where did you saw that I said something about i686 *kernel*? I think that I explicitly mentioned *x86_64* kernel with i686 userland and described why it could be beneficial for some users with limited memory.

As for security, I don't think that running your own computer in a tightly controlled environment should be *that* dangerous. At least many users did it for years without problems. That looks like a scare. In either case it's the user who should decide what's best for him. I don't think that educated grown-up people should be treated like babies.

Other distributions might drop it or not, we'll see. At least Debian is not dropping it yet. But this is a moot point now. After all those discussions I see that nobody really cares about user interests here. At least in Debian's case they stated that their users interests is of utmost priority to them in contrast to just useless technical innovation. And I'm not a proponent of consumerism. So take it lightly.

On Sun, Sep 8, 2019 at 7:00 AM vvs vvs vvs009@gmail.com wrote:

I'm sorry, but where did you saw that I said something about i686 *kernel*? I think that I explicitly mentioned *x86_64* kernel with i686 userland and described why it could be beneficial for some users with limited memory.

As for security, I don't think that running your own computer in a tightly controlled environment should be *that* dangerous. At least many users did it for years without problems. That looks like a scare. In either case it's the user who should decide what's best for him. I don't think that educated grown-up people should be treated like babies.

Other distributions might drop it or not, we'll see. At least Debian is not dropping it yet. But this is a moot point now. After all those discussions I see that nobody really cares about user interests here. At least in Debian's case they stated that their users interests is of utmost priority to them in contrast to just useless technical innovation. And I'm not a proponent of consumerism. So take it lightly.

You know, rather than complaining, you could revive the x86 SIG and help us support it?

Richard Stallman, like many others, described freedom of choice insofar as freedom to contribute and use. He often uses the phrases "helping your neighbor" or "share with your neighbor", and this is a core aspect of Free Software.

None of us will turn away assistance for maintaining i686 userland if there is truly interest in it. It's being retired because no one stepped up in the past 18 months to help maintain it.

Even I, who needs it in some circumstances, am already spread so thinly that I cannot help with i686-specific issues very much, so I did not become a member of the x86 SIG. But if this is something that is important to you, then *please* help to restore it.

Proper architecture support does not come without effort, and no one has been supplying any.

I promise you, *all* major Linux distributions, and many minor ones have been having variations of this conversation. Expect to see more distributions dropping 32-bit x86 support over the next year or so as people stop helping to maintain the architecture.

-- 真実はいつも一つ！/ Always, there's only one truth!

On Sun, Sep 8, 2019 at 7:00 AM vvs vvs vvs009@gmail.com wrote:

I'm sorry, but where did you saw that I said something about i686 *kernel*? I think that I explicitly mentioned *x86_64* kernel with i686 userland and described why it could be beneficial for some users with limited memory.

And a child with no vaccines isn't that much of a danger to the community. Until someone opens a window, or forgets to wash a glass, and the *other* unvaccinated children form a wonderful way to spread an infection.

As for security, I don't think that running your own computer in a tightly controlled environment should be *that* dangerous. At least many users did it for years without problems. That looks like a scare. In either case it's the user who should decide what's best for him. I don't think that educated grown-up people should be treated like babies.

There is "deciding what's best for him". There is also "supporting them inem in an unmaintainable and unsecurable environment, at the expense of disk space and mirror space and build time and bugzilla resources for a dangerously obsolete architecture.

Other distributions might drop it or not, we'll see. At least Debian is not dropping it yet. But this is a moot point now. After all those discussions I see that nobody really cares about user interests here. At least in Debian's case they stated that their users interests is of utmost priority to them in contrast to just useless technical innovation. And I'm not a proponent of consumerism. So take it lightly.

The heck? If you know of others who need or want i686 architecture for reasons other than theoretical completeness, please encourage them to speak up. And offer to do some of the work.

On Sun, Sep 8, 2019 at 11:44 PM John M. Harris Jr. johnmh@splentity.com wrote:

On Sunday, September 8, 2019 7:05:39 PM MST Nico Kadel-Garcia wrote:

...

On Sun, Sep 8, 2019 at 7:00 AM vvs vvs vvs009@gmail.com wrote:

...

I'm sorry, but where did you saw that I said something about i686 *kernel*? I think that I explicitly mentioned *x86_64* kernel with i686 userland and described why it could be beneficial for some users with limited memory.

And a child with no vaccines isn't that much of a danger to the community. Until someone opens a window, or forgets to wash a glass, and the *other* unvaccinated children form a wonderful way to spread an infection.

I fail to see how a comparison of x86 to a disease is really relevant. If you really want to go that route, I'll tell you now that x86 is no more "unsecurable" than x64, both arches have security issues, and it's unavoidable. There are also vulnerabilities with specific implementations.

You snipped the part about how few if any people, including primary software authors, are supporting x86 architectures anymore, so they will not be maintained or patched. Thus, they will not be protected against attacks, even obsolete attacks. If there are more than a few such machines, and they are not completely isolated, i.e. they are permitted to play with the vaccinated children, they can form pools of vulnerable hosts and defeat the "herd immunity" from the generally better maintained and supported software.

It's the pool of unmaintained and unmaintainable software I'm expressing a concern about. If you'd care to take on the task of supporting the architecture, of porting patches and testing them, well, the world has many freedoms and this is one of them. I just don't think you'll have many clients as the architecture has almost universally been discarded. There are few tools, like FreeDOS and wine, for running quite obsolete software, that may still benefit from the availability of x86 libraries in Fedora. Maybe even Xen? But not many.

On Mon, Sep 09, 2019 at 02:52:43AM -0700, John M. Harris Jr. wrote:

While I don't have statistics on that, all of the anecdotal evidence supports that vendors are still "supporting" 32 bit software. In fact, nearly all proprietary software for RHEL, for example, is 32 bit.

The EDA tools I'm forced to use dropped support for 32-bit systems in 2015. Even before that, I recall my own testing showing that the 64-bit versions were noticably faster, thanks to architectural improvements inherent to x86_64.

That's now how vulnerabilities work, and just being 64 bit doesn't solve any security issue.

All else being equal, it allows ASLR to be considerably more effective.

- Solomon

On Monday, September 9, 2019 4:43:18 AM MST Vitaly Zaitsev via devel wrote:

On 9/9/19 11:52 AM, John M. Harris Jr. wrote:

...

That's now how vulnerabilities work, and just being 64 bit doesn't solve any security issue.

https://en.wikipedia.org/wiki/Address_space_layout_randomization

ASLR has nothing to do with the wild claims made in that email, that having an x86 system will somehow taint or 'infect' other systems. Additionally, you don't need to run a 64 bit system to get ASLR.

On Monday, September 9, 2019 5:16:23 AM MST Vitaly Zaitsev via devel wrote:

On 9/9/19 1:47 PM, John M. Harris Jr. wrote:

...

ASLR has nothing to do with the wild claims made in that email, that having an

x86 system will somehow taint or 'infect' other systems.

...

Additionally, you don't need to run a 64 bit system to get ASLR.

i686 app has only 4 GB of virtual address space (2 GB for user-space and 2 GB for kernel space), so ASLR is ineffective.

-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org)

The system I'm sending this email from only has 4 GiB of memory in total. Does that mean that this system makes ASLR completely ineffective? Should this arch also be removed from Fedora, because of that?

So, if I'd start to use Debian i686 instead of Fedora or will use ARM32 device instead of ARM64 the world will be a safer place? Also, I was told that maintaining i686 Fedora code base myself would be fine, but in the same time I'm told that it's not acceptable from the safety point of view. Why I'm smelling a contradiction here? In short: the decision to drop i686 support is supported by contradicting statements, while at the same time if I want to be taken seriously, I should bring strong evidence. That's very objective discussion.

* John M. Harris, Jr.:

ASLR has nothing to do with the wild claims made in that email, that having an x86 system will somehow taint or 'infect' other systems. Additionally, you don't need to run a 64 bit system to get ASLR.

I'm not saying that the analogy is appropriate, but it is just not true that 32-bit support is isolated. We will have to patch lots of code to support the new *_time64 system calls, and that will impact everyone, even for applications that are never run on 32-bit systems.

(This assumes that we can magically fix the native linker issues on 32-bit architectures.)

Thanks, Florian

On Saturday, September 7, 2019 11:44:59 AM MST Victor V. Shkamerda wrote:

I totally agree with that view. Making such decisions without public discussion is not respecting user's freedom of choice. And this list doesn't count as a public discussion. Nobody will know about it outside a very closed circle. If you don't know exact numbers or reasons why people still use that architecture, then rushing to drastic measures just won't have enough rationale and will be viewed as a lack of care.

The reasons to use 32-bit userland might be more than you think. There are different use scenarios besides browsing Internet or virtualization. There could be a high cost of upgrade (think about replacing many computers at once). There could be a unique peripheral equipment. There could be compatibility concerns. And there could be just not enough memory.

OTOH, what could justify removing repository entirely? Why not make it an option which could be installed on demand? That would be better than leave everyone out on the cold. Is the cost to build it really that high? Safety could not be a reason in a tightly contained environment. Let users decide for themselves.

There is no black and white distinction: multilib or i686 kernel. There are reasons why using x86_64 kernel with i686 userland might be a better option. Some older office computers were manufactured with 64-bit CPU but without support for more than 4 GB RAM. In such case using i686 userland on a x86_64 kernel provides much more free memory than using 64-bit userland. Such configuration is unsupported, but neither is i686. Giving users a choice is always better than decide for them behind their backs.

And of course there is still an option to switch to another OS. Do I need to remind that Linux and Red Hat were not created just to replace some other OS, but to respect freedom of choice? What happened that this is not even mentioned in such discussions? Is this just business as usual?

Wait, what happened to x86 becoming a secondary architecture? You know, there are vendors that still create and sell x86 systems today.

Ok, if that's so hard then I'm apologize for not recognizing the pain.

OTOH, if Debian has resources to maintain the support for at least next five years it means one of two things: either they have more resources than Fedora, or something is wrong with your assessment.

I'd help with maintaining 32-bit userland as much as I can. But I'm afraid that's not much. From my point of view the only support I need is that damn thing worked most of the time. And there no more bugs in i686 userland than in x86_64 one. If you really need so much patching than I simply don't understand why it still works on other supported 32-bit arches all over the world, e.g. ARM.

P.S. And what it's all supposed to do with "Linux is NOT about choice"? This looks like just as an excuse to me for some other thing.

I'm happy with any support no matter how it is defined. In fact I didn't get very much support from Fedora either over more than 20 years, so my expectations are quite low.

If there is something more relevant than freedom of choice, then there is no point arguing further, because I value community relations over any technical reasons.

No I didn't, but I must be sure that you speak on behalf of everyone before making my choices.

Thanks for the suggestion. But I'm sure that I don't need so much bureaucracy just to run my little errands. If that's how Fedora is operated, than it won't make much difference for me to just using another distribution.

BTW, that just means that Fedora is refusing to provide much needed services even to a people who are ready to accept most of that support burden themselves and I'm one of them.

And why people are not reading all the answers? That was a rhethorical question.

I said it already several times, that I don't need volunteers to fix things for me! I just need an already built repository which I could just use and fix things myself if needed. But Fedora is refusing to provide such repository which was built automatically by Koji. I don't care if something was broken in that repository as long as I can access those binaries and fix them if needed.

But no matter how frequently I repeat it, I'm always get answers that I'm insisting that somebody should work for me. No I'm not. If it's so difficult to keep that repository around, then it's just fine with me and I'll find another way. But that would be very unfortunate.

Oh, brother...

So, you are insisting that Koji just doesn't work without any assistance? And that it's impossible to build a separate i686 repository without affecting all others? And that you can't exclude that architecture for a specific package? If that's the case then it's very different from what I already knew.

I didn't answered your other question because I've answered the same question several times already. Yes, I have a use cases where I'll get a severe performance hit if I was not careful. And this is related to available memory and swapping. And I can't afford losing yet another hundred megabytes for no particular reason. And I don't think that constantly upgrading my computer is the answer. I remember times when it was possible to install Red Hat Linux on a computer with 32 MB of RAM. Going in that direction I should do nothing but upgrade every now and then even though I don't want my computer to affect my activities that hard. And some people thought that Windows 95 was a memory hog!

Honestly, I'm tired. And I don't think that arguing further will be of any help. I'm convinced that I really have no other choice but just use some Linux distribution that doesn't requires me to spend so much time explaining to everyone why I need things in my life to be the way I need it.

And it reminds me about some guy who maintains some quite popular software. He used some old SUSE until he was forced to upgrade. After struggling with consequences he just bought Apple computer with Mac OS. He explained that he is just too old to waste so much time on unimportant things. I don't remember his name.

On 9/9/19 3:35 PM, vvs vvs wrote:

I didn't answered your other question because I've answered the same question several times already. Yes, I have a use cases where I'll get a severe performance hit if I was not careful. And this is related to available memory and swapping. And I can't afford losing yet another hundred megabytes for no particular reason. And I don't think that constantly upgrading my computer is the answer. I remember times when it was possible to install Red Hat Linux on a computer with 32 MB of RAM. Going in that direction I should do nothing but upgrade every now and then even though I don't want my computer to affect my activities that hard. And some people thought that Windows 95 was a memory hog!

In all your emails, you have never said that you've tried to use a 64-bit userspace. You keep making this claim that if you did you would lose "another hundred megabytes", but have you actually tried it?

You can still run Linux in an extremely limited amount of RAM. I do it all the time with openwrt. But do you really want to go back to the level of functionality you had back then?

On 9/10/19 7:55 AM, vvs vvs wrote:

Did I? I thought that I've said that I'm using x86_64 kernel right now and that I have my memory stretched to the limits already.

But yes, I've experimented with x86_64 userland some time ago, I don't remember exact numbers but I think that I've lost 100-200 MB of memory. And I have not much time to experiment every time something have changed.

You are right that I can reduce memory footprint by carefully tuning my system. But that's what I've already did and Fedora breaks it just too often. I'm used to work without GNOME on my other computer that have only 32-bit CPU. But it started to be very painful to upgrade to newer Fedora releases. So, when I've got 64-bit computer I've just went with the flow and don't customize parts I'm not really interested in. I want to spend on it as little time as possible. Going for my own Linux from scratch is just not viable.

Wait---so you are using 32-bit Gnome on a 64-bit capable CPU running 64-bit kernel? If the reason is to save 200MB of memory, you should definitely try one of the memory-thrifty desktop environments like xfce.

You also said that you're running a memory-hungry non-Fedora application, which you presumably compile yourself.

Therefore, it looks to me that you should try a 64-bit Fedora with a memory-saving xfce, install a 32-bit GCC toolchain and compile your app in 32-bit mode.

On Tuesday, September 10, 2019 12:41:14 PM MST Przemek Klosowski via devel wrote:

On 9/10/19 7:55 AM, vvs vvs wrote:

...

Did I? I thought that I've said that I'm using x86_64 kernel right now and that I have my memory stretched to the limits already.

...

But yes, I've experimented with x86_64 userland some time ago, I don't remember exact numbers but I think that I've lost 100-200 MB of memory. And I have not much time to experiment every time something have changed.

...

You are right that I can reduce memory footprint by carefully tuning my system. But that's what I've already did and Fedora breaks it just too often. I'm used to work without GNOME on my other computer that have only 32-bit CPU. But it started to be very painful to upgrade to newer Fedora releases. So, when I've got 64-bit computer I've just went with the flow and don't customize parts I'm not really interested in. I want to spend on it as little time as possible. Going for my own Linux from scratch is just not viable.

Wait---so you are using 32-bit Gnome on a 64-bit capable CPU running 64-bit kernel? If the reason is to save 200MB of memory, you should definitely try one of the memory-thrifty desktop environments like xfce.

You also said that you're running a memory-hungry non-Fedora application, which you presumably compile yourself.

Therefore, it looks to me that you should try a 64-bit Fedora with a memory-saving xfce, install a 32-bit GCC toolchain and compile your app in 32-bit mode.

Compiling his app as 32 bit would require 32 bit repositories for the libraries he plans on linking.

- - John Harris

On 9/10/19 11:48 PM, drago01 wrote:

On Wednesday, September 11, 2019, John M. Harris Jr. <johnmh@splentity.com mailto:johnmh@splentity.com> wrote: Compiling his app as 32 bit would require 32 bit repositories for the libraries he plans on linking.

Multilib is still supported so libraries are present in the repositories.

And to be really clear about it, those 32-bit multilib libraries are in the 64-bit repository, so for now they aren't going away even if the 32-bit repo does.

I did test some of these desktops in the past. From my experience LXDT should be just fine. Anyway, thanks for reminding me, because I was so used to standard Fedora desktop that completely forgot about such alternatives.

You are welcome to use the koji buildroot repo for that. https://kojipkgs.fedoraproject.org/repos/f30-build/latest

Thanks. That would be just splendid, but won't it cease to exist after Fedora 30 EOL? Then it's just a temporary workaround.

In the interests of not making this thread a bunch longer, I am just going to answer a number of things here in one place.

On 9/7/19 11:44 AM, Victor V. Shkamerda wrote:

I totally agree with that view. Making such decisions without public discussion is not respecting user's freedom of choice. And this list doesn't count as a public discussion. Nobody will know about it outside a very closed circle. If you don't know exact numbers or reasons why people still use that architecture, then rushing to drastic measures just won't have enough rationale and will be viewed as a lack of care.

There was a lot of discussion on this list, in fesco tickets in fesco meetings, on phoronix, etc.

I'm not sure what you mean by 'respecting users freedom of choice'. We cannot possibly provide all choices that anyone wants or thinks they want. Really it comes down to (in rough order of effectiveness):

* Try and convince people doing the work to provide/continue to provide the thing you want, but realize that the people doing the work are under no obligation here, you need to convince them there is some reason they find compelling.

* Offer to do some / part / all of the work, but realize here too you need to convince the people doing the work now that it's worth the time / resources to allow you to do the work (although this is a much better 'sell' than just convincing people.

It's pretty clear that i686 is dwindling as an arch. It was pretty clear a few years ago when it was demoted to a alternative arch and the x86 sig was setup to try and work on issues that came up. No one really did so, so it's time to take the next steps.

What work should be done? Please, be more specific. Right now I'm running a i686 userland and it works. If I would be able to build the whole repository myself I'm pretty sure that most things will still work. If it won't work I might try to fix it and contribute patches back. But without that repository I can't even try it in the first place.

Lets step back a step here. Why are you running a 32bit userspace? There's not really any advantage (and some disadvantages) to doing so.

The koji buildroot repo will continue to be available if you want to copy something, but as far as work to be done to move back to distributing a i686 set of trees? I guess doing the release blocking tests on i686 at Beta and Final might be a good start, but thats a ton of work for one person... is there anyone else you have talked to that wants to do this?

kevin

On 9/9/19 12:47 PM, vvs vvs wrote:

I don't even know anyone whom I could address. I'm already spent too much time on that list trying to convince everyone that I'm ready to take all the burden of using unsupported packages, but was told that it's against Fedora policies. What much could I do?

Having read the thread, you seem to miss the point that's been repeatedly made: the packages occasionally fail to build, and someone has to fix them.  That act, fixing packages when they don't build is the "support" that someone has to provide.

You can't use packages that don't exist.  They don't exist unless someone supports them.  Therefore you can't use an unsupported package.  It's not because policy forbids it, it's because they don't exist without the act of a human maintainer making them build (which is described as "supporting" the package.)

Does that make sense?

On 9/9/19 12:47 PM, vvs vvs wrote:

Having read the thread, you seem to miss the point that's been repeatedly made: the packages occasionally fail to build, and someone has to fix them.  That act, fixing packages when they don't build is the "support" that someone has to provide.

Koji still builds the i686 packages by default even on F32

You can't use packages that don't exist.  They don't exist unless someone supports them.  Therefore you can't use an unsupported package.  It's not because policy forbids it, it's because they don't exist without the act of a human maintainer making them build (which is described as "supporting" the package.)

Does that make sense?

No!

On Tuesday, September 10, 2019 3:23:29 PM CEST Miroslav Suchý wrote:

Dne 09. 09. 19 v 21:01 Kevin Fenzi napsal(a):

...

The koji buildroot repo will continue to be available if you want to copy something, but as far as work to be done to move back to distributing a i686 set of trees? I guess doing the release blocking tests on i686 at Beta and Final might be a good start, but thats a ton of work for one person... is there anyone else you have talked to that wants to do this?

I want to state one consequence. As there is no compose, the mock configs fedora-31-i386 and fedora-rawhide-i386 will point directly to Koji.

https://github.com/rpm-software-management/mock/commit/a0c5d493c362c993d6961...

All local builds into this chroot will likely be slow. And I will likely remove (or move to /etc/mock/eol/ ) those files in near future.

This may affect CI of 3rd parties.

FTR, we changed this in Copr as well - otherwise _all_ the builds in fedora-31-i386+ would already fail. So please note that the fedora 31+ `i386` build chroot are _different_ from other architectures (in copr/mock), and really re-consider whether you want to build against them.

Pavel

On Mon, Jul 15, 2019 at 11:52:50AM +0200, Mark Wielaard wrote:

Could you explain a bit more how this (keeps) working? I think my mental model of how Fedora repositories work in the case of multilib devel packages is a bit flawed. At first I assumed that this suggestion would kill that. Because I was under the impression that it worked by having the i386 repository be part of the x86_64 package repository. So removing the i686 repositories would mean removing all 32bit packages from x86_64. But from your explanation above it seems that is not how it works. So if there are no i686 repositories, then where/how do the 32bit multilib packages come from?

I think I am mixing up buildroots, koji targets, repositories in my mind.

Let me second that request. It would be great if somebody could summarize the steps that lead to 32bit packages being installable on amd64 machines. I know how the beginning (compilation in the appropriate chroot), and the end (dnf install foo.i686), but the middle is hazy.

For example I maintain two packages that provide 32bit multilib binaries. They are produced by the i686 koji target. Then elfutils- devel.i686 (to create 32bit multilib binaries) and valgrind.i686 (to run 32bit multilib binaries) are provided to the user because the i686 repository is part of the x86_64 repository (if I understand things correctly). Will this just keep working, or will I have to make changes to my packages to keep this working after the disappearance of the i686 repository?

Zbyszek

On 7/15/19 8:08 AM, Tom Hughes wrote:

On 15/07/2019 15:56, Zbigniew Jędrzejewski-Szmek wrote:

...

On Mon, Jul 15, 2019 at 11:52:50AM +0200, Mark Wielaard wrote:

...

Could you explain a bit more how this (keeps) working? I think my mental model of how Fedora repositories work in the case of multilib devel packages is a bit flawed. At first I assumed that this suggestion would kill that. Because I was under the impression that it worked by having the i386 repository be part of the x86_64 package repository. So removing the i686 repositories would mean removing all 32bit packages from x86_64. But from your explanation above it seems that is not how it works. So if there are no i686 repositories, then where/how do the 32bit multilib packages come from?

I think I am mixing up buildroots, koji targets, repositories in my mind.

Let me second that request. It would be great if somebody could summarize the steps that lead to 32bit packages being installable on amd64 machines. I know how the beginning (compilation in the appropriate chroot), and the end (dnf install foo.i686), but the middle is hazy.

It definitely doesn't work as Mark thinks because only a subset of the i686 packages are present in the x86_64 repository.

I've never really been sure how it decides what packages to include but certainly the idea here that every 32 bit package that you might need on a 64 bit system is present is false because I actually have a list of additional packages that get copied over in my local mirror ;-)

So, this is how it works (as far as I recall off the top of my head):

You build a archfull package in koji. It's built for x86_64 and i686 (and the other arches).

pungi runs to compose things. It has a config (in pungi-fedora or bodhi config) that tells it what to do for multilib. It has also config that tells it what tag(s) to use for the compose, so it talks to koji and asks about/gathers packages based on that tag.

Currently for rawhide that is:

multilib = [ ('^Everything$', { 'x86_64': ['devel', 'runtime'], }) ]

# format: {arch|*: [packages]} multilib_blacklist = { '*': ['kernel', 'kernel-PAE*', 'kernel*debug*', 'dmraid-devel', 'kdeutils-devel', 'mkinitrd-devel', 'php-devel', 'java-*', 'httpd-devel', 'tomcat-native', 'php*', 'httpd', 'krb5-server', 'krb5-server-ldap', 'mod_*', 'ghc-*' ], }

# format: {arch|*: [packages]} multilib_whitelist = { '*': ['libgnat', 'wine', 'lmms-vst', 'nspluginwrapper', 'libflashsupport', 'valgrind', 'perl-libs', 'redhat-lsb', 'yaboot', 'syslinux-extlinux-nonlinux', 'syslinux-nonlinux', 'syslinux-tftpboot', 'nosync', '*-static', 'apitrace-libs', 'fakeroot-libs', 'postgresql-odbc', 'mysql-connector-odbc', 'fakechroot-libs','mesa-vdpau-drivers', 'p11-kit-trust', 'mariadb-connector-odbc', 'compiler-rt', 'nvidia-query-resource-opengl-lib', 'ibus-libs', 'ibus-gtk2', 'ibus-gtk3', 'glib-networking' ], }

So, that says to use the 'runtime' method of multilib, and then explicitly add some packages and explicitly remove some. The orig idea of 'runtime' was to allow runtime for i686 binaries on x86_64 platforms.

Pungi uses in turn the python-multilib package to do the actual computations here, which you can see in: https://pagure.io/releng/python-multilib/blob/master/f/multilib/multilib.py#...

Basically it looks to see if a package has files in specific directories (mostly runtime libraries) and returns a true or false if the package should be multilibed. There is a bunch of cruft and corner cases in there as well.

pungi then puts those i686 packages that are marked true into the x86_64 tree and creates the repo. x86_64 users can then install those i686 packages easily because they are in the x86_64 repos they already use.

None of the above needs a public, mirrored i686 repository tree. The computation is done by pungi at compose time looking at all the i686 packages built in koji and tagged with the appropriate tag.

Did that help? Or make things worse... happy to try and expand or answer any other questions about it.

kevin

Hello, John Reiser.

Sun, 14 Jul 2019 14:35:46 -0700 you wrote:

For some apps 2GB of malloc() arena is plenty, and they run faster in 32-bit mode because a 64-byte cache line contains 16 pointers instead of only 8.

And such applications became extremely vulnerable due to missing ASLR support. Also they cannot allocate and use more than 2 GB of RAM.

One year after Red Hat Enterprise Linux version 7 reaches end-of-support.

Fedora's development cycle is not related to RHEL's.

-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org)

On 7/14/19 11:46 PM, Florian Weimer wrote:

• Miro Hrončok:

...

The only other use/need for the repostories is to allow maintainers to debug and test fixes for multilib shipped packages, but the koji buildroot repo can be used for this use case.

...

** modify mock to use the koji buildroot for i686 for f31+ for those few users that need to build i686 packages locally.

Due to lack of debuginfo packages in Koji, this results in a worse user experience. (See https://pagure.io/koji/issue/540.)

Yeah, that is a downside. :( Last updated a year ago... wonder where it is on the roadmap. Will ask.

I also think all architectures should behave consistently in mock. It's odd that after the proposed change, only on i686, mock will be affected by buildroot overrides in the default configuration, for instance.

Also true.

I just don't think the number of people who do local i686 builds is all that large, so it having some issues and corner cases to help out the vast majority of folks seems like a good trade off to me.

kevin

On 7/15/19 3:11 PM, Kevin Kofler wrote:

Kevin Fenzi wrote:

...

I just don't think the number of people who do local i686 builds is all that large, so it having some issues and corner cases to help out the vast majority of folks seems like a good trade off to me.

Removing something does not "help out" anybody, ever. The people who'd better not use the repo can just opt to not use it. You don't have to delete it under them. And those who will use it probably have a good reason to use it.

This argument is absurd. Removing something definitely helps out the people who were producing it, if they no longer wish to spend resources making it. If the people using it wish it to stay around, they would have to take over producing it.

kevin