On 01/26/2009 07:04, Ulrich Drepper wrote:
from a last year message:
Jakub Jelinek wrote:
The koji build boxes all run RHEL 5. Getting them upgraded to a not-yet- released kernel seems unlikely.
I know it is a pain, on the other hand it would really improve Fedora 11.
Not only that. It is the only way to actually test what we are shipping.
At least from glibc's POV (but indirectly from a much wider range) we have to compile everything on the kernel we are shipping for the release. Period. I know that the current build infrastructure doesn't do this but this only means it has to change. We have virtualization available, there is no excuse.
Is there any plan to build fedora with their own kernel ?
It's a 'must have', for some packages.
e.g. GLIBC: http://sourceware.org/git/?p=glibc.git;a=blob_plain;f=sysdeps/unix/sysv/linu...
A lot of features can't be used, because 2.6.18(3.5 years old) is the base kernel.
On Sat, May 15, 2010 at 04:38:02PM +0200, Xose Vazquez Perez wrote:
On 01/26/2009 07:04, Ulrich Drepper wrote:
from a last year message:
Jakub Jelinek wrote:
The koji build boxes all run RHEL 5. Getting them upgraded to a not-yet- released kernel seems unlikely.
I know it is a pain, on the other hand it would really improve Fedora 11.
Not only that. It is the only way to actually test what we are shipping.
At least from glibc's POV (but indirectly from a much wider range) we have to compile everything on the kernel we are shipping for the release. Period. I know that the current build infrastructure doesn't do this but this only means it has to change. We have virtualization available, there is no excuse.
Is there any plan to build fedora with their own kernel ?
It's a 'must have', for some packages.
e.g. GLIBC: http://sourceware.org/git/?p=glibc.git;a=blob_plain;f=sysdeps/unix/sysv/linu...
A lot of features can't be used, because 2.6.18(3.5 years old) is the base kernel.
I'm confused. Is the glibc build process testing for features on the _currently running kernel_ in some fashion, or is it testing for features in headers provided by the kernel-headers package (which is a proper BR already)? I would expect the latter, meaning the currently running kernel doesn't really matter.
On Sat, May 15, 2010 at 04:38:02PM +0200, Xose Vazquez Perez wrote:
On 01/26/2009 07:04, Ulrich Drepper wrote:
from a last year message:
Jakub Jelinek wrote:
The koji build boxes all run RHEL 5. Getting them upgraded to a not-yet- released kernel seems unlikely.
I know it is a pain, on the other hand it would really improve Fedora 11.
Not only that. It is the only way to actually test what we are shipping.
At least from glibc's POV (but indirectly from a much wider range) we have to compile everything on the kernel we are shipping for the release. Period. I know that the current build infrastructure doesn't do this but this only means it has to change. We have virtualization available, there is no excuse.
Is there any plan to build fedora with their own kernel ?
It's a 'must have', for some packages.
e.g. GLIBC: http://sourceware.org/git/?p=glibc.git;a=blob_plain;f=sysdeps/unix/sysv/linu...
A lot of features can't be used, because 2.6.18(3.5 years old) is the base kernel.
And it causes a load of obscure Koji-specific bugs which require workarounds just for Koji builds. This was the latest one:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg25242.html
Hopefully those builders can be upgraded to RHEL 6 soon ...
Rich.
On 05/16/2010 05:31 AM, Richard W.M. Jones wrote:
Is there any plan to build fedora with their own kernel ?
It's a 'must have', for some packages.
e.g. GLIBC: http://sourceware.org/git/?p=glibc.git;a=blob_plain;f=sysdeps/unix/sysv/linu...
A lot of features can't be used, because 2.6.18(3.5 years old) is the base kernel.
And it causes a load of obscure Koji-specific bugs which require workarounds just for Koji builds. This was the latest one:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg25242.html
Hopefully those builders can be upgraded to RHEL 6 soon ...
Rich.
Unless RHEL 6 is needed to get sufficient VM capability, as was said above would it not be cleaner to compile the entire release in a VM using fedora itself?
Perhaps around alpha release time, clean and rebuild everything with itself from scratch and then keep updating that release and using it to build itself.
gene/
On Sun, May 16, 2010 at 09:06:52AM -0400, Genes MailLists wrote:
Unless RHEL 6 is needed to get sufficient VM capability, as was said above would it not be cleaner to compile the entire release in a VM using fedora itself?
Last I heard, the builders were going to be changed to use a VM to fully encapsulate the build. Don't know what the timelines were on this. There was some talk of using libguestfs to upload and/or download files into the VMs which is why I heard about this.
The problem with using a VM created using the latest unstable Fedora is that the latest unstable Fedora might not be working. Perhaps we could always build Fedora n+1 on top of Fedora n.
Rich.
On 05/17/2010 04:59 AM, Richard W.M. Jones wrote:
The problem with using a VM created using the latest unstable Fedora is that the latest unstable Fedora might not be working. Perhaps we could always build Fedora n+1 on top of Fedora n.
Rich.
What some do is keep several build trees - at least 2 - (say B0, B1, B2 ) - let current rawhide be B2.
use B2 to build B3 - if it succeeds then move current rawhide to B3 otherwise it remains at B2.
I wasn't sure if your N and N-1 above are builds or released versions - but I think we're on the same page.
gene
On Mon, May 17, 2010 at 09:35:37AM -0400, Genes MailLists wrote:
On 05/17/2010 04:59 AM, Richard W.M. Jones wrote:
The problem with using a VM created using the latest unstable Fedora is that the latest unstable Fedora might not be working. Perhaps we could always build Fedora n+1 on top of Fedora n.
Rich.
What some do is keep several build trees - at least 2 - (say B0, B1, B2 ) - let current rawhide be B2.
use B2 to build B3 - if it succeeds then move current rawhide to B3 otherwise it remains at B2.
Just because you manage to build a kernel, or even manage to boot a kernel, doesn't mean it really works. We really need all the testing that goes into a full Fedora release before we trust what is built from that, which is why my suggestion of building Fedora N+1 on Fedora N.
Rich.
On 05/17/2010 11:22 AM, Richard W.M. Jones wrote:
otherwise it remains at B2.
Just because you manage to build a kernel, or even manage to boot a kernel, doesn't mean it really works. We really need all the testing that goes into a full Fedora release before we trust what is built from that, which is why my suggestion of building Fedora N+1 on Fedora N.
Yes of course = by "it succeeds" I meant passes regression tests etc.
On Mon, 2010-05-17 at 11:41 -0400, Genes MailLists wrote:
On 05/17/2010 11:22 AM, Richard W.M. Jones wrote:
otherwise it remains at B2.
Just because you manage to build a kernel, or even manage to boot a kernel, doesn't mean it really works. We really need all the testing that goes into a full Fedora release before we trust what is built from that, which is why my suggestion of building Fedora N+1 on Fedora N.
Yes of course = by "it succeeds" I meant passes regression tests etc.
We don't by any means have enough programmed testing to be confident about this kind of status for more than milestone releases at present. Given that, it certainly seems much safer to just build F(N+1) on F(N), as proposed earlier.
On 05/17/2010 07:15 PM, Adam Williamson wrote:
We don't by any means have enough programmed testing to be confident about this kind of status for more than milestone releases at present. Given that, it certainly seems much safer to just build F(N+1) on F(N), as proposed earlier.
Yes - tho it might be good to build itself once we hit RC or something.
Course its tempting to avoid any changes like that too as we come round the corner and see the finish line ...
On Mon, May 17, 2010 at 09:59:08AM +0100, Richard W.M. Jones wrote:
On Sun, May 16, 2010 at 09:06:52AM -0400, Genes MailLists wrote:
Unless RHEL 6 is needed to get sufficient VM capability, as was said above would it not be cleaner to compile the entire release in a VM using fedora itself?
Last I heard, the builders were going to be changed to use a VM to fully encapsulate the build. Don't know what the timelines were on this. There was some talk of using libguestfs to upload and/or download files into the VMs which is why I heard about this.
I haven't heard this and it might be a misinterpretation of other things.
skvidal and I are working on a builder for third party add on repositories for Fedora. That builder would be separate from koji and would, indeed use a vm for building. You might want to talk to mbonnet if there's plans on adding vm capability to koji. (I don't think there is as that was one of the reasons we decided we should be creating a different build system).
-Toshio
On Mon, May 17, 2010 at 11:17:11AM -0400, Toshio Kuratomi wrote:
On Mon, May 17, 2010 at 09:59:08AM +0100, Richard W.M. Jones wrote:
On Sun, May 16, 2010 at 09:06:52AM -0400, Genes MailLists wrote:
Unless RHEL 6 is needed to get sufficient VM capability, as was said above would it not be cleaner to compile the entire release in a VM using fedora itself?
Last I heard, the builders were going to be changed to use a VM to fully encapsulate the build. Don't know what the timelines were on this. There was some talk of using libguestfs to upload and/or download files into the VMs which is why I heard about this.
I haven't heard this and it might be a misinterpretation of other things.
skvidal and I are working on a builder for third party add on repositories for Fedora. That builder would be separate from koji and would, indeed use a vm for building. You might want to talk to mbonnet if there's plans on adding vm capability to koji. (I don't think there is as that was one of the reasons we decided we should be creating a different build system).
Fair enough, I was confused then.
Rich.
Toshio Kuratomi wrote:
skvidal and I are working on a builder for third party add on repositories for Fedora. That builder would be separate from koji and would, indeed use a vm for building.
Yet another one? We already have Plague and Koji, what would the new one offer over those? Would it be easier to set up?
Kevin Kofler
On 05/16/2010 11:31 AM, Richard W.M. Jones wrote:
And it causes a load of obscure Koji-specific bugs which require workarounds just for Koji builds. This was the latest one:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg25242.html
Hopefully those builders can be upgraded to RHEL 6 soon ...
The snake biting its tail.
Fedora should be self-built.
On Sun, 23 May 2010, Xose Vazquez Perez wrote:
On 05/16/2010 11:31 AM, Richard W.M. Jones wrote:
And it causes a load of obscure Koji-specific bugs which require workarounds just for Koji builds. This was the latest one:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg25242.html
Hopefully those builders can be upgraded to RHEL 6 soon ...
The snake biting its tail.
Fedora should be self-built.
rhel6 is based on fedora.
So, we are self-built.
-sv
On Sun, 23 May 2010, Xose Vazquez Perez wrote:
On 05/16/2010 11:31 AM, Richard W.M. Jones wrote:
And it causes a load of obscure Koji-specific bugs which require workarounds just for Koji builds. This was the latest one:
http://www.mail-archive.com/qemu-devel@nongnu.org/msg25242.html
Hopefully those builders can be upgraded to RHEL 6 soon ...
The snake biting its tail.
Fedora should be self-built.
I don't think anyone thinks rebuilding the builders every 6 months (and fixing all the bugs that comes with that) is a good use of their time.
-Mike
On 05/23/2010 02:25 PM, Mike McGrath wrote:
I don't think anyone thinks rebuilding the builders every 6 months (and fixing all the bugs that comes with that) is a good use of their time.
-Mike
But a release really should be boot strapped with itself once when it gets stable enough .. or at least built with Fedora N-1 as someone else suggested.
I sort of recall someone mumbling about using a VM but I don't know what would be involved in keeping the vm updated to Fedora N-1 ?
On Sun, May 23, 2010 at 17:21:36 -0400, Genes MailLists lists@sapience.com wrote:
On 05/23/2010 02:25 PM, Mike McGrath wrote:
I don't think anyone thinks rebuilding the builders every 6 months (and fixing all the bugs that comes with that) is a good use of their time.
-Mike
But a release really should be boot strapped with itself once when it gets stable enough .. or at least built with Fedora N-1 as someone else suggested.
I sort of recall someone mumbling about using a VM but I don't know what would be involved in keeping the vm updated to Fedora N-1 ?
People already ready run scripts to look for FTBFS packages now. Having to deal with all of these at once would be a real problem. Doing this this way allows us to spread out the fixing over time.
On Sun, May 23, 2010 at 04:45:49PM -0500, Bruno Wolff III wrote:
People already ready run scripts to look for FTBFS packages now. Having to deal with all of these at once would be a real problem. Doing this this way allows us to spread out the fixing over time.
The problem isn't FTBFS packages, but packages that require devious workarounds because Koji isn't quite like any other distro out there.
Koji is a Fedora userspace running on top of a RHEL 5 kernel. This tests out all sorts of strange and interesting paths inside glibc, because glibc emulates a lot of system calls which didn't exist in RHEL 5. While extra testing is usually great, it's not clear that dumping this testing on to package maintainers is a good thing, particularly since Koji is about the only real world case where people actually run mismatched userspace and kernel like this. (There is no distro that I know of that advocates running a brand new userspace on top of an old kernel -- but please correct me if I'm wrong about that).
Rich.
On Mon, 2010-05-24 at 13:01 +0100, Richard W.M. Jones wrote:
On Sun, May 23, 2010 at 04:45:49PM -0500, Bruno Wolff III wrote:
People already ready run scripts to look for FTBFS packages now. Having to deal with all of these at once would be a real problem. Doing this this way allows us to spread out the fixing over time.
The problem isn't FTBFS packages, but packages that require devious workarounds because Koji isn't quite like any other distro out there.
Koji is a Fedora userspace running on top of a RHEL 5 kernel. This tests out all sorts of strange and interesting paths inside glibc, because glibc emulates a lot of system calls which didn't exist in RHEL 5. While extra testing is usually great, it's not clear that dumping this testing on to package maintainers is a good thing, particularly since Koji is about the only real world case where people actually run mismatched userspace and kernel like this. (There is no distro that I know of that advocates running a brand new userspace on top of an old kernel -- but please correct me if I'm wrong about that).
While building in vms is starting to make sense on x86, it still doesn't make sense (or not possible) on other arches like ppc, s390, arm, sparc, ia64, etc... So building for those arches is going to continue to use the mock path, which means if we don't do it for x86, we're going to run into these odd code paths on even "odder" arches and make keeping those up and running even harder.
On Mon, May 24, 2010 at 01:14:07PM -0700, Jesse Keating wrote:
On Mon, 2010-05-24 at 13:01 +0100, Richard W.M. Jones wrote:
On Sun, May 23, 2010 at 04:45:49PM -0500, Bruno Wolff III wrote:
People already ready run scripts to look for FTBFS packages now. Having to deal with all of these at once would be a real problem. Doing this this way allows us to spread out the fixing over time.
The problem isn't FTBFS packages, but packages that require devious workarounds because Koji isn't quite like any other distro out there.
Koji is a Fedora userspace running on top of a RHEL 5 kernel. This tests out all sorts of strange and interesting paths inside glibc, because glibc emulates a lot of system calls which didn't exist in RHEL 5. While extra testing is usually great, it's not clear that dumping this testing on to package maintainers is a good thing, particularly since Koji is about the only real world case where people actually run mismatched userspace and kernel like this. (There is no distro that I know of that advocates running a brand new userspace on top of an old kernel -- but please correct me if I'm wrong about that).
While building in vms is starting to make sense on x86, it still doesn't make sense (or not possible) on other arches like ppc, s390, arm, sparc, ia64, etc... So building for those arches is going to continue to use the mock path, which means if we don't do it for x86, we're going to run into these odd code paths on even "odder" arches and make keeping those up and running even harder.
http://www.linux-kvm.org/page/PowerPC http://www.mjmwired.net/kernel/Documentation/ia64/kvm.txt http://www.mjmwired.net/kernel/Documentation/s390/kvm.txt http://sourceforge.net/projects/arm-kvm/ + some mail-list disscussion about KVM on SPARC.
It is certainly possible. Often in experimental state, but nevertheless. We have the technology for VMs.
On Mon, 2010-05-24 at 23:31 +0200, Tomasz Torcz wrote:
It is certainly possible. Often in experimental state, but nevertheless. We have the technology for VMs.
Right, sounds perfect for running a build system under...
On Mon, May 24, 2010 at 11:31:09PM +0200, Tomasz Torcz wrote:
On Mon, May 24, 2010 at 01:14:07PM -0700, Jesse Keating wrote:
On Mon, 2010-05-24 at 13:01 +0100, Richard W.M. Jones wrote:
On Sun, May 23, 2010 at 04:45:49PM -0500, Bruno Wolff III wrote:
People already ready run scripts to look for FTBFS packages now. Having to deal with all of these at once would be a real problem. Doing this this way allows us to spread out the fixing over time.
The problem isn't FTBFS packages, but packages that require devious workarounds because Koji isn't quite like any other distro out there.
Koji is a Fedora userspace running on top of a RHEL 5 kernel. This tests out all sorts of strange and interesting paths inside glibc, because glibc emulates a lot of system calls which didn't exist in RHEL 5. While extra testing is usually great, it's not clear that dumping this testing on to package maintainers is a good thing, particularly since Koji is about the only real world case where people actually run mismatched userspace and kernel like this. (There is no distro that I know of that advocates running a brand new userspace on top of an old kernel -- but please correct me if I'm wrong about that).
While building in vms is starting to make sense on x86, it still doesn't make sense (or not possible) on other arches like ppc, s390, arm, sparc, ia64, etc... So building for those arches is going to continue to use the mock path, which means if we don't do it for x86, we're going to run into these odd code paths on even "odder" arches and make keeping those up and running even harder.
http://www.linux-kvm.org/page/PowerPC http://www.mjmwired.net/kernel/Documentation/ia64/kvm.txt http://www.mjmwired.net/kernel/Documentation/s390/kvm.txt http://sourceforge.net/projects/arm-kvm/
- some mail-list disscussion about KVM on SPARC.
It is certainly possible. Often in experimental state, but nevertheless. We have the technology for VMs.
Actually, we don't for PowerPC. Right now, you have two options for virt on ppc/ppc64:
1) kvm for PPC 440. Fedora doesn't support PPC 440 (it could, but it's an embedded CPU and would require yet another kernel.)
2) LPARs for ppc64. This is actually a bit more feasible, but it requires you to have a POWER machine with LPAR support configured. Most of the existing builders are blades, which generally don't do LPARs.
josh
Genes MailLists lists@sapience.com writes:
On 05/23/2010 02:25 PM, Mike McGrath wrote:
I don't think anyone thinks rebuilding the builders every 6 months (and fixing all the bugs that comes with that) is a good use of their time.
But a release really should be boot strapped with itself once when it gets stable enough .. or at least built with Fedora N-1 as someone else suggested.
I still carry the scars from when mysql stopped building in brew, a couple years ago. Not just rawhide, but brew in general. Eventually (after a couple months of bewilderment) it emerged that the build machines had been migrated from RHEL4 to RHEL5 kernels, and on the PPC machines that entailed a change of page size, and mysql had an obscure dependency on the page size. So even though brew was building inside version-specific buildroots, things broke.
So I'm firmly on the side of "do not change the build machines' kernels often". Every time you do it, you not only risk breaking the current branch builds, but past branches. The consequences for reproducibility and wasted developer time are enormous.
What we need is not "fedora N is built on fedora N", but "we know what fedora N is built on, and it *won't* change midstream". The notion of rebasing the build machines at some point approaching release is beyond stupid.
I could maybe buy "fedora N is built on fedora N-1", but there are two problems with that: first, you'd need a separate build farm for every branch, and second, fedora N-1 is still a moving target for much of the life of fedora N.
regards, tom lane
On Sun, May 23, 2010 at 11:11:57PM -0400, Tom Lane wrote:
I could maybe buy "fedora N is built on fedora N-1", but there are two problems with that: first, you'd need a separate build farm for every branch,
With mock running directly inside Xen VMs as is the case now, then yes. But with mock running inside a tiny VM which is just launched for the purpose of the build, you shouldn't need extra build farms.
and second, fedora N-1 is still a moving target for much of the life of fedora N.
Well, this gets back to the updates policy :-)
The issue is what kernel does Fedora N-1 have, and that should hopefully be a bit more stable.
Rich.
On 05/23/2010 11:11 PM, Tom Lane wrote:
I still carry the scars from when mysql stopped building in brew, a couple years ago. Not just rawhide, but brew in general. Eventually (after a couple months of bewilderment) it emerged that the build
Frustrating I am sure - but it could have been the other way round - built fine and didnt work on the newer kernel ... or whatever.
Or what about a developer adding new features to mysql in your example - they would find themselves unable to compile as well quite possibly.
I think a better rule is - if there are problems, let us reasonably find and fix them.
The dont-touch-it-it-may-break philosophy is not obviously a better policy in all situations.
gene/
On Sun, 2010-05-23 at 17:21 -0400, Genes MailLists wrote:
On 05/23/2010 02:25 PM, Mike McGrath wrote:
I don't think anyone thinks rebuilding the builders every 6 months (and fixing all the bugs that comes with that) is a good use of their time.
-Mike
But a release really should be boot strapped with itself once when it gets stable enough .. or at least built with Fedora N-1 as someone else suggested.
I sort of recall someone mumbling about using a VM but I don't know what would be involved in keeping the vm updated to Fedora N-1 ?
It is built with itself, all except the running kernel behind the mock chroot.
Xose Vazquez Perez xose.vazquez@gmail.com writes:
It's a 'must have', for some packages.
e.g. GLIBC: http://sourceware.org/git/?p=glibc.git;a=blob_plain;f=sysdeps/unix/sysv/linu...
A lot of features can't be used, because 2.6.18(3.5 years old) is the base kernel.
This is not true. It only means that some features must be tested for at runtime, instead of just assuming they are present. The only effect is that glibc is slightly larger.
Andreas.
devel@lists.stg.fedoraproject.org
-
Adam Williamson -
Andreas Schwab -
Bruno Wolff III -
Genes MailLists -
Jesse Keating -
Josh Boyer -
Kevin Kofler -
Matt Domsch -
Mike McGrath -
Richard W.M. Jones -
Seth Vidal -
Tom Lane -
Tomasz Torcz -
Toshio Kuratomi -
Xose Vazquez Perez