This is very much a draft, but I'd like to start a discussion regarding what we expect from primary architectures. Feedback not only welcome, but actively desired.
-------------------------------------------------------------------------
Secondary architectures in Fedora are subject to looser constraints than primary architectures for two primary reasons:
1) To make it easier to bootstrap an architecture without the overhead of the primary architecture release engineering process 2) To avoid primary architecture development being held up by poorly developed or niche architectures
Promoting an architecture to primary architecture status is a significant responsibility. It implies that the port is sufficiently mature that little in the way of further architecture-specific changes or rebuilds will be required, and also that it has enough development effort to avoid it delaying the development of other primary architectures. Further, it means that the architecture becomes part of the overall Fedora brand. Fedora is an integrated Linux distribution rather than a technology collection, and as such there are various expectations that the overall Fedora experience will be consistent over all primary architectures.
In order to ensure that these expectations are met, secondary architectures must meet various criteria before they can be promoted:
1) There must be adequate representation for the architecture on the Fedora infrastructure and release engineering teams. 2) All builds must occur on Fedora-maintained build servers. 3) Where technically possible, all supported hardware targets must be supported via Anaconda. Exceptions are limited to highly resource constrained devices or hardware which provides no means for simultaneous support of install and target media. 4) All supported platforms must have kernels built from the Fedora kernel SRPM and enabled by default in the spec file. Each kernel must be built in a timely manner for every SRPM upload. 5) Sufficient developer resources must be available to fix any architecture-specific issues in such a way that they do not delay overall Fedora development. 6) It must be possible for maintainers of critical-path hardware dependent packages to have direct access to supported hardware in order to rectify any release-blocking issues. For example, X maintainers must have direct access to any hardware with graphics capabilities. 7) The port must not rely on sourceless binaries unless they fall under the generic firmware exemption. Where source and toolchain are available, the binaries must be built in the Fedora build infrastructure.
As such, promotion to primary architecture status will require agreement from the Fedora infrastructure, release engineering, kernel and installer teams and is subject to overall approval by the Fedora Steering Committee.
On Tue, Mar 20, 2012 at 03:19:35PM +0000, Matthew Garrett wrote:
This is very much a draft, but I'd like to start a discussion regarding what we expect from primary architectures. Feedback not only welcome, but actively desired.
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Jakub
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line? No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
On Tue, Mar 20, 2012 at 10:58 AM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line? No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
Actually, I hadn't thought about it that way before, but having a build fail on x86 or x86_64 would allow me to kill the ARM build and save load on the ARM buildsys. A win, if it's going to fail anyway.
-J
-- Brendan Conoboy / Red Hat, Inc. / blc@redhat.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Tue, Mar 20, 2012 at 11:58 AM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line? No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
I thought about this a bit yesterday. My conclusions are that it will impact people in 2 cases.
1) The build works fine on i686 and x86_64 and completes in the current "normal" time, but then the ARM build fails some number of minutes/hours later. For most packages, that likely isn't a big deal but for large packages like gcc and the kernel, I could have done exactly what you propose and downloaded the x86 results while waiting hours for ARM to complete and tested. If the ARM build fails while I'm doing that, I need to go and redo that testing after ARM is fixed because it failing will fail the entire koji build.
2) Updates. Submitting updates requires the entire build to be complete which means you have to wait for the slowest thing to finish. Having to wait for 12 hours effectively means you can't even test your update until the next day, and then after you test it you can submit it. Again, this is mostly compounded for large packages, but it does impact workflow.
josh
----- Original Message -----
From: "Josh Boyer" jwboyer@gmail.com To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Cc: "Jakub Jelinek" jakub@redhat.com, secondary@lists.fedoraproject.org Sent: Tuesday, 20 March, 2012 4:08:16 PM Subject: Re: RFC: Primary architecture promotion requirements
On Tue, Mar 20, 2012 at 11:58 AM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line? No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
I thought about this a bit yesterday. My conclusions are that it will impact people in 2 cases.
- The build works fine on i686 and x86_64 and completes in the
current "normal" time, but then the ARM build fails some number of minutes/hours later. For most packages, that likely isn't a big deal but for large packages like gcc and the kernel, I could have done exactly what you propose and downloaded the x86 results while waiting hours for ARM to complete and tested. If the ARM build fails while I'm doing that, I need to go and redo that testing after ARM is fixed because it failing will fail the entire koji build.
- Updates. Submitting updates requires the entire build to be
complete which means you have to wait for the slowest thing to finish. Having to wait for 12 hours effectively means you can't even test your update until the next day, and then after you test it you can submit it. Again, this is mostly compounded for large packages, but it does impact workflow.
3) chain builds in rawhide become slower.
For X we do a lot of chain + dependency building, and I think the gnome guys do as well
So now I have 12 packages to update, I need the first two to complete before I can get the next 10 etc,
Dave.
On Tue, 2012-03-20 at 12:08 -0400, Josh Boyer wrote:
- Updates. Submitting updates requires the entire build to be complete
which means you have to wait for the slowest thing to finish. Having to wait for 12 hours effectively means you can't even test your update until the next day, and then after you test it you can submit it. Again, this is mostly compounded for large packages, but it does impact workflow.
From a QA perspective, there's a fairly important instance of this case.
We sometimes wind up working on very compressed timescales in validation sprints. We don't get very long to do validation.
So it's not unusual for me to be bugging, say, the kernel team to give us a new kernel build that fixes a blocker bug, so we can do a new release candidate, so we can test the release candidate in twelve hours, so we can make the go/no-go meeting deadline the next morning.
If builds get significantly slower, that could have a concrete impact on the release validation process: it's plausible that we'd either need to extend the validation period somewhat - earlier freezes - or we would have to eat a somewhat higher likelihood of release slippages.
On Tue, Mar 20, 2012 at 11:46 PM, Adam Williamson awilliam@redhat.com wrote:
On Tue, 2012-03-20 at 12:08 -0400, Josh Boyer wrote:
- Updates. Submitting updates requires the entire build to be complete
which means you have to wait for the slowest thing to finish. Having to wait for 12 hours effectively means you can't even test your update until the next day, and then after you test it you can submit it. Again, this is mostly compounded for large packages, but it does impact workflow.
From a QA perspective, there's a fairly important instance of this case. We sometimes wind up working on very compressed timescales in validation sprints. We don't get very long to do validation.
So it's not unusual for me to be bugging, say, the kernel team to give us a new kernel build that fixes a blocker bug, so we can do a new release candidate, so we can test the release candidate in twelve hours, so we can make the go/no-go meeting deadline the next morning.
If builds get significantly slower, that could have a concrete impact on the release validation process: it's plausible that we'd either need to extend the validation period somewhat - earlier freezes - or we would have to eat a somewhat higher likelihood of release slippages.
Thanks Adam, this is the first real use case where speed of builds is important for something other than keeping the developer happy.
Peter
On 03/21/2012 06:26 AM, Peter Robinson wrote:
Thanks Adam, this is the first real use case where speed of builds is important for something other than keeping the developer happy.
Other points raised on the list are:
1. The nature of chainbuilds would feel slowed build times particularly. This is more of a multi-developer happy item.
2. Total turnaround time on security updates.
On Tue, Mar 20, 2012 at 08:58:45AM -0700, Brendan Conoboy wrote:
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line? No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
Looking at last gcc build times (not unusual, though I really remember arm taking much longer than that, e.g. 4.7.0-0.11.fc17 took almost 17 hours on both arm architectures), from http://*koji.fedoraproject.org/packages/gcc/4.7.0/0.20.fc17/data/logs/*/stat... : i686 4h18m x86_64 1h25m ppc 4h12m ppc64 4h16m s390 6h27m s390x 6h04m armv5tel 26h20m armv7hl 24h17m
So even speeding this up twice means it is still 2x slower than the slowest other secondary architecture.
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
For the builds completed on some architectures, but waiting on others nothing is moved over to the packages/ dirs. Yes, you can grab them from the task directories, but only manually, scripts fetching testsuite results won't see them, it can't be filed into bodhi, in rawhide isn't tagged into the buildroots, etc.
Jakub
Jakub Jelinek wrote:
Looking at last gcc build times (not unusual, though I really remember arm taking much longer than that, e.g. 4.7.0-0.11.fc17 took almost 17 hours on both arm architectures), from http://*koji.fedoraproject.org/packages/gcc/4.7.0/0.20.fc17/data/logs/*/stat... : i686 4h18m x86_64 1h25m ppc 4h12m ppc64 4h16m s390 6h27m s390x 6h04m armv5tel 26h20m armv7hl 24h17m
So even speeding this up twice means it is still 2x slower than the slowest other secondary architecture.
Ouch!!!
That shows that ARM should be the LAST architecture we consider for primary status rather than the first. (That said, I don't think it makes sense to make PPC primary again or to make S/390 primary. They don't have anywhere near the market share. But IMHO ARM doesn't have the market share either.)
Kevin Kofler
Kevin Kofler wrote:
But IMHO ARM doesn't have the market share either.
Kevin, you don't know what you are talking about. Every cell phone has an ARM cpu in it. Smart phone or otherwise. Almost every HDTV has an ARM cpu in it. Almost every tablet has an ARM cpu in it. What do people buy these days? Phones, tablets, and TVs. Not desktop computers. Hell, ARM is even building server boxes now (this is probably where Red Hat's interest is in).
I'll enjoy calling up these emails 2 years from now when you're complaining that Fedora isn't ready for ARM (if we don't start now).
Michael Cronenworth wrote:
Kevin, you don't know what you are talking about. Every cell phone has an ARM cpu in it. Smart phone or otherwise. Almost every HDTV has an ARM cpu in it. Almost every tablet has an ARM cpu in it.
Several of those are not suitable devices to run a general purpose GNU/Linux distribution on, and even for those which are, why would they be our primary target?
What do people buy these days? Phones, tablets, and TVs. Not desktop computers.
Citation needed. Desktop/notebook computers aren't going to go away any time soon.
Hell, ARM is even building server boxes now
But even those servers are not fast enough to complete package builds in a reasonable time.
(this is probably where Red Hat's interest is in).
As far as I know, this proposal is driven by community people, not Red Hat people.
I'll enjoy calling up these emails 2 years from now when you're complaining that Fedora isn't ready for ARM (if we don't start now).
We're starting now, that's what the secondary architecture is for. There's no need for ARM to be a primary architecture for Fedora to be "ready" for it.
(And I don't see myself using an ARM as my primary machine in 2 years. It's more likely that I'll still be using the same machines I use now, I don't replace my hardware that often, and even this old Pentium 4 is faster than a "fast" ARM machine.)
Kevin Kofler
On 03/20/2012 10:54 AM, Kevin Kofler wrote:
As far as I know, this proposal is driven by community people, not Red Hat people.
Many people in the Fedora ARM community are Red Hat people, but that's hardly relevant to the proposal.
We're starting now, that's what the secondary architecture is for. There's no need for ARM to be a primary architecture for Fedora to be "ready" for it.
No, Fedora ARM started years ago. There comes a point where a secondary cannot continue to grow. To become more than it is, it needs the support and interest of the main Fedora community. We are reaching that point. That is why we are having this discussion.
(And I don't see myself using an ARM as my primary machine in 2 years. It's more likely that I'll still be using the same machines I use now, I don't replace my hardware that often, and even this old Pentium 4 is faster than a "fast" ARM machine.)
I sincerely doubt it. Compare specs to a Tegra 3 chip. And that's just a mobile system.
On 3/20/12 11:18 AM, Brendan Conoboy wrote:
We're starting now, that's what the secondary architecture is for. There's no need for ARM to be a primary architecture for Fedora to be "ready" for it.
No, Fedora ARM started years ago. There comes a point where a secondary cannot continue to grow. To become more than it is, it needs the support and interest of the main Fedora community. We are reaching that point. That is why we are having this discussion.
Honestly I've yet to see a succinct list of reasons why secondary arch is no longer good enough for the ARM effort, for at least the next few releases. I may have missed it in the flurry of emails and debate, anybody care to recap it for clarity?
On 03/20/2012 11:20 AM, Jesse Keating wrote:
Honestly I've yet to see a succinct list of reasons why secondary arch is no longer good enough for the ARM effort, for at least the next few releases. I may have missed it in the flurry of emails and debate, anybody care to recap it for clarity?
This was one of the points raised by FESCo yesterday, and it's a fine question that we'll be answering better, elsewhere, in due course. That said, where does this question lead? If we explain what we're trying to get to, will it somehow overcome the objections raised such as build system performance? For the sake of coherent discussion, let's assume that we have good reasons why we want to move to primary, and we can keep the subject on what the requirements are for doing so. The topic at hand isn't even ARM specific, it's just been prompted by us ARM aficionados. Again, I understand that there do need to be good reasons, that's just not the subject of this particular thread. So, other than build system performance, what are the requirements you'd like to see met?
On 3/20/12 11:50 AM, Brendan Conoboy wrote:
On 03/20/2012 11:20 AM, Jesse Keating wrote:
Honestly I've yet to see a succinct list of reasons why secondary arch is no longer good enough for the ARM effort, for at least the next few releases. I may have missed it in the flurry of emails and debate, anybody care to recap it for clarity?
This was one of the points raised by FESCo yesterday, and it's a fine question that we'll be answering better, elsewhere, in due course. That said, where does this question lead? If we explain what we're trying to get to, will it somehow overcome the objections raised such as build system performance? For the sake of coherent discussion, let's assume that we have good reasons why we want to move to primary, and we can keep the subject on what the requirements are for doing so. The topic at hand isn't even ARM specific, it's just been prompted by us ARM aficionados. Again, I understand that there do need to be good reasons, that's just not the subject of this particular thread. So, other than build system performance, what are the requirements you'd like to see met?
Knowing the reasons you want to upgrade to PA is important because it will help us judge whether or not the cost of the upgrade is worth the result, or whether or not the result could be obtained while still staying SA.
I don't think promoting a SA to PA is something that can be generically covered, each such potential action needs to be looked at, discussed, weighed, measured, etc... To know whether or not we as a project should absorb the cost of promoting ARM to PA, we need to know what the benefit is, or what the expected benefit would be.
As for the other requirements, I believe there are enough sub-threads hashing that out :)
Brendan Conoboy wrote:
This was one of the points raised by FESCo yesterday, and it's a fine question that we'll be answering better, elsewhere, in due course. That said, where does this question lead? If we explain what we're trying to get to, will it somehow overcome the objections raised such as build system performance? For the sake of coherent discussion, let's assume that we have good reasons why we want to move to primary, and we can keep the subject on what the requirements are for doing so. The topic at hand isn't even ARM specific, it's just been prompted by us ARM aficionados. Again, I understand that there do need to be good reasons, that's just not the subject of this particular thread.
It doesn't make sense to discuss requirements for becoming a primary architecture without discussing whether it should be considered in the first place. I don't see ANY reasons why it's needed. And as I wrote in my first reply in this thread, I don't think there should be a generic process for becoming a primary architecture at all, it should be a change done only in very exceptional cases.
Kevin Kofler
On Tue, 20 Mar 2012 20:14:14 +0100 Kevin Kofler kevin.kofler@chello.at wrote:
It doesn't make sense to discuss requirements for becoming a primary architecture without discussing whether it should be considered in the first place. I don't see ANY reasons why it's needed. And as I wrote in my first reply in this thread...
<snip>
Yes, we know your position. No need to continue to reply to every email in the thread repeating it. If you have some new info to add by all means do so.
Thanks.
kevin
On Mar 20, 2012, at 1:14 PM, Kevin Kofler wrote:
It doesn't make sense to discuss requirements for becoming a primary architecture without discussing whether it should be considered in the first place.
Seems requirements are needed to have the discussion, to have metrics based rather than emotion based consideration. Actively pushing that adrenaline button as a significant means of making decisions might be entertaining, but eventually a real conversation is the goal.
Chris Murphy
On Tue, Mar 20, 2012 at 06:54:07PM +0100, Kevin Kofler wrote:
Michael Cronenworth wrote:
Kevin, you don't know what you are talking about. Every cell phone has an ARM cpu in it. Smart phone or otherwise. Almost every HDTV has an ARM cpu in it. Almost every tablet has an ARM cpu in it.
Several of those are not suitable devices to run a general purpose GNU/Linux distribution on, and even for those which are, why would they be our primary target?
It's a matter of time, and not very much time at that.
My £400 tablet has plenty enough power, storage and whatever else to run Fedora. Fedora works pretty well on £200 Trim Slice servers. Fedora is going to be shipped with £25 Raspberry Pi devices in the near future.
Rich.
On Tue, Mar 20, 2012 at 7:54 PM, Kevin Kofler kevin.kofler@chello.atwrote:
What do people buy these days? Phones, tablets, and TVs. Not desktop computers.
Citation needed. Desktop/notebook computers aren't going to go away any time soon.
http://www.economist.com/node/21531109 with some interesting charts
On Thu, 2012-03-22 at 12:57 +0200, Nikos Roussos wrote:
On Tue, Mar 20, 2012 at 7:54 PM, Kevin Kofler kevin.kofler@chello.at wrote: > What do people buy these days? Phones, tablets, and TVs. Not desktop > computers.
Citation needed. Desktop/notebook computers aren't going to go away any time soon.
http://www.economist.com/node/21531109 with some interesting charts
Which is actually confirmation of the previous sentence. The smartphones, tablets and similar devices together are growing much much faster than desktops and notebooks and are already much bigger market but they are not replacement for the desktops and notebooks.
Which also supports the idea of having Fedora support both of these groups of computing devices well.
On Thu, Mar 22, 2012 at 11:27 AM, Tomas Mraz tmraz@redhat.com wrote:
On Thu, 2012-03-22 at 12:57 +0200, Nikos Roussos wrote:
On Tue, Mar 20, 2012 at 7:54 PM, Kevin Kofler kevin.kofler@chello.at wrote: > What do people buy these days? Phones, tablets, and TVs. Not desktop > computers.
Citation needed. Desktop/notebook computers aren't going to go away any time soon.
http://www.economist.com/node/21531109 with some interesting charts
Which is actually confirmation of the previous sentence. The smartphones, tablets and similar devices together are growing much much faster than desktops and notebooks and are already much bigger market but they are not replacement for the desktops and notebooks.
That depends, in the developed western world probably not, in the developing world most users are just going straight to smartphones and tablets and not bothering with desktops/laptops at all. The reasons for this is low power and price. In those areas desktops are dead and the increase in usage in this markets is 100s of millions of devices are year and for a lot of users it's their only means of accessing the internet and associated information.
Which also supports the idea of having Fedora support both of these groups of computing devices well.
Exactly one of the primary drivers for us to open up the discussion.
Peter
On Mar 22, 2012, at 5:33 AM, Peter Robinson wrote:
That depends, in the developed western world probably not, in the developing world most users are just going straight to smartphones and tablets and not bothering with desktops/laptops at all. The reasons for this is low power and price.
Exactly. Power, price, infrastructure, space.
Chris Murphy
Once upon a time, Michael Cronenworth mike@cchtml.com said:
Kevin, you don't know what you are talking about. Every cell phone has an ARM cpu in it. Smart phone or otherwise. Almost every HDTV has an ARM cpu in it. Almost every tablet has an ARM cpu in it. What do people buy these days? Phones, tablets, and TVs. Not desktop computers.
And how many cell phones, tablets, and TVs is Fedora ARM planning to target? It doesn't sound like that's the target market (at least at this time).
On 03/20/2012 11:39 AM, Chris Adams wrote:
And how many cell phones, tablets, and TVs is Fedora ARM planning to target? It doesn't sound like that's the target market (at least at this time).
Indeed, targeting mobile devices on day 1 is beyond the scope of the proposal. The first step is the eat-our-own-dogfood target, which is self-hosting ARM servers. Mobile devices are a natural direction for Fedora ARM, of course, but as with every new direction there are a different set of challenges to be worked through. For now we're just talking about the core OS.
Once upon a time, Brendan Conoboy blc@redhat.com said:
Indeed, targeting mobile devices on day 1 is beyond the scope of the proposal. The first step is the eat-our-own-dogfood target, which is self-hosting ARM servers. Mobile devices are a natural direction for Fedora ARM, of course, but as with every new direction there are a different set of challenges to be worked through. For now we're just talking about the core OS.
Okay, but how many ARM servers are in widespread use? For the resources required as a primary arch, there should be a large expected user base. The first sentence of the detailed description on the feature page is "ARM processors are the most popular CPUs in the world." but that is entirely irrelevant if you aren't targeting a significant number of those CPUs.
On 03/20/2012 11:50 AM, Chris Adams wrote:
Okay, but how many ARM servers are in widespread use? For the resources required as a primary arch, there should be a large expected user base. The first sentence of the detailed description on the feature page is "ARM processors are the most popular CPUs in the world." but that is entirely irrelevant if you aren't targeting a significant number of those CPUs.
Believe me, I want to target those CPUs, but no single proposal can include all the steps necessary to get there. Think of ARM-on-primary as the first of many steps designed to get us there. If you've ever climbed a mountain you'll know that the trick getting to the top is to put one foot in front of the other. This is just a step along the way.
On 3/20/12 11:54 AM, Brendan Conoboy wrote:
Believe me, I want to target those CPUs, but no single proposal can include all the steps necessary to get there. Think of ARM-on-primary as the first of many steps designed to get us there. If you've ever climbed a mountain you'll know that the trick getting to the top is to put one foot in front of the other. This is just a step along the way.
I hate analogies, but no, the first step is working out in a gym to make sure you're in fit enough shape to go up the mountain.
SA is your gym, and from what I've seen in these threads I really don't think ARM is ready for the mountain yet.
On Mar 20, 2012, at 1:03 PM, Jesse Keating wrote:
I hate analogies, but no, the first step is working out in a gym to make sure you're in fit enough shape to go up the mountain.
As a distractor from long, heated threads, and mountain person - gym bunnies get to altitude and implode routinely. If you want to be in shape for mountains, climb lots of stairs. I guess gyms have stair masters. Even dinky hills around here in Aspen get the heart to 90% MHR easily. So you need to work at getting to and maintaining high heart rates. Advance acclimation helps also.
But this *requirements* thread is about acclimation, planning and anticipating the challenges of the climb. Serious climbs may involve days or months of this. So if the analogy holds, a lot of advance work has to be done before ARM actually is promoted to primary. This isn't the go, no-go meeting, or gear-up time. This is the shopping list.
Now the ultra ridiculous: How about secondary architecture requirements demoted as-is to tertiary. And create substantially more aggressive requirements for secondary architecture (in which ARM would be placed), yet are not identical requirements to primary architecture requirements?
Chris Murphy
On 03/20/2012 12:44 PM, Chris Murphy wrote:
But this *requirements* thread is about acclimation, planning and anticipating the challenges of the climb. Serious climbs may involve days or months of this. So if the analogy holds, a lot of advance work has to be done before ARM actually is promoted to primary. This isn't the go, no-go meeting, or gear-up time. This is the shopping list.
Thank you :-)
Now the ultra ridiculous: How about secondary architecture requirements demoted as-is to tertiary. And create substantially more aggressive requirements for secondary architecture (in which ARM would be placed), yet are not identical requirements to primary architecture requirements?
Yes, the all-or-nothing mindset between secondary and primary is almost certainly the root of the problem. We want more representation in Fedora than being a secondary connotes, but at the same time, ARM today does not fulfill every aspect of what PA means. The discussion is about what is required, how to get there. There has to be a way to represent Fedora's architectural support as a spectrum of features, not simply a binary assessment. I absolutely do want to get ARM onto the same koji build system, but don't want to make ARM the mortal enemy of every packager whose workflow is materially hampered. This discussion has been quite useful in shaping my opinion that we need to improve the koji portion of the proposal (Principally because of chain builds), but how to make those improvements needs to be worked out.
On Mar 20, 2012, at 1:52 PM, Brendan Conoboy wrote:
Yes, the all-or-nothing mindset between secondary and primary is almost certainly the root of the problem.
Well that and I think there's some resistance at the notion that for the massive consumer market, the desktop is a dead platform. Not quite letter type dead. Yet. But it basically just experienced a drive by shooting in 2011, and it's just a matter of time before the patient bleeds out.
I actually find mobile and tablet quite irritating as a platform, presently. Nevertheless, desktop is rapidly becoming more of an anchor, than a mountain to be climbed.
Chris Murphy
On Tue, Mar 20, 2012 at 12:52:58PM -0700, Brendan Conoboy wrote:
On 03/20/2012 12:44 PM, Chris Murphy wrote:
Now the ultra ridiculous: How about secondary architecture requirements demoted as-is to tertiary. And create substantially more aggressive requirements for secondary architecture (in which ARM would be placed), yet are not identical requirements to primary architecture requirements?
Yes, the all-or-nothing mindset between secondary and primary is almost certainly the root of the problem. We want more representation in Fedora than being a secondary connotes,
Maybe you should begin by convincing package maintainers that ARM is a good platform (if it is; I do not know) and that they want to use it, instead of (figuratively speaking) shoving it down their throats by means of FESCo decision to promote ARM to primary arch. If you attract enough people, the transition may just happen "naturally"...
D.
On Wed, Mar 21, 2012 at 7:13 AM, David Tardon dtardon@redhat.com wrote:
On Tue, Mar 20, 2012 at 12:52:58PM -0700, Brendan Conoboy wrote:
On 03/20/2012 12:44 PM, Chris Murphy wrote:
Now the ultra ridiculous: How about secondary architecture requirements demoted as-is to tertiary. And create substantially more aggressive requirements for secondary architecture (in which ARM would be placed), yet are not identical requirements to primary architecture requirements?
Yes, the all-or-nothing mindset between secondary and primary is almost certainly the root of the problem. We want more representation in Fedora than being a secondary connotes,
Maybe you should begin by convincing package maintainers that ARM is a good platform (if it is; I do not know) and that they want to use it, instead of (figuratively speaking) shoving it down their throats by means of FESCo decision to promote ARM to primary arch. If you attract enough people, the transition may just happen "naturally"...
There is no doubt it is a good platform, it's not about shoving it down people's throats, it's about making Fedora available on millions of devices that are cheap and low power. The transition is happening already and it happening due to cost and power, whether that be in the data centre running on servers or in the developing world. You just have to look at the millions of ARM based devices being sold in China, India and other parts of Asia.
Peter
On Tue, 2012-03-20 at 13:44 -0600, Chris Murphy wrote:
Now the ultra ridiculous: How about secondary architecture requirements demoted as-is to tertiary. And create substantially more aggressive requirements for secondary architecture (in which ARM would be placed), yet are not identical requirements to primary architecture requirements?
Yes, this might be actually the best short-term path. Or rather than demoting the other secondary architectures to tertiary status just have different scale for various secondary architectures in terms of official infrastructure support. I can see automatic spawning of secondary builds for ARM in the main koji instance, use of main bodhi, etc., etc.
Once upon a time, Brendan Conoboy blc@redhat.com said:
Believe me, I want to target those CPUs, but no single proposal can include all the steps necessary to get there. Think of ARM-on-primary as the first of many steps designed to get us there. If you've ever climbed a mountain you'll know that the trick getting to the top is to put one foot in front of the other. This is just a step along the way.
Okay, but why is ARM-as-primary-arch an early step, and not near the end? Increasing the developer and engineering burden across the whole project should not be done for a small target audience.
On 03/20/2012 12:03 PM, Chris Adams wrote:
Okay, but why is ARM-as-primary-arch an early step, and not near the end? Increasing the developer and engineering burden across the whole project should not be done for a small target audience.
Really there is no beginning and no end, so we're somewhere in the middle ;-) ARM-as-secondary was earlier. ARM-as-primary is next. Fedora-on-tablets is later. Fedora-on-cellphones is later. The bottom line is that Fedora is an rpm-based native-built operating system, so ARM servers come first. Fedora isn't currently built to run efficiently and smoothly on embedded devices. That's okay, it's nice to have followup projects. Meanwhile ARM servers are going to be important too.
Brendan Conoboy wrote:
On 03/20/2012 12:03 PM, Chris Adams wrote:
Okay, but why is ARM-as-primary-arch an early step, and not near the end? Increasing the developer and engineering burden across the whole project should not be done for a small target audience.
Really there is no beginning and no end, so we're somewhere in the middle ;-) ARM-as-secondary was earlier. ARM-as-primary is next. Fedora-on-tablets is later. Fedora-on-cellphones is later. The bottom line is that Fedora is an rpm-based native-built operating system, so ARM servers come first. Fedora isn't currently built to run efficiently and smoothly on embedded devices. That's okay, it's nice to have followup projects. Meanwhile ARM servers are going to be important too.
You haven't answered his question: why would ARM-as-primary come before Fedora-on-tablets and Fedora-on-cellphones? Those can be perfectly supported using the secondary architecture infrastructure (or if not, we need to improve that infrastructure).
Kevin Kofler
On 03/20/2012 03:33 PM, Kevin Kofler wrote:
You haven't answered his question: why would ARM-as-primary come before Fedora-on-tablets and Fedora-on-cellphones? Those can be perfectly supported using the secondary architecture infrastructure (or if not, we need to improve that infrastructure).
That's two questions. I'll answer the first one: ARM Server semiconductors are more amenable to open source licensing than embedded & telecommunication companies, so getting a complete open source ARM Server running is simple compared to getting an ARM Linux tablet or cell phone running.
On Tue, 2012-03-20 at 13:50 -0500, Chris Adams wrote:
Once upon a time, Brendan Conoboy blc@redhat.com said:
Indeed, targeting mobile devices on day 1 is beyond the scope of the proposal. The first step is the eat-our-own-dogfood target, which is self-hosting ARM servers. Mobile devices are a natural direction for Fedora ARM, of course, but as with every new direction there are a different set of challenges to be worked through. For now we're just talking about the core OS.
Okay, but how many ARM servers are in widespread use? For the resources required as a primary arch, there should be a large expected user base. The first sentence of the detailed description on the feature page is "ARM processors are the most popular CPUs in the world." but that is entirely irrelevant if you aren't targeting a significant number of those CPUs.
...yet, is the word you're missing, which changes things quite a bit. What the ARM project is currently targeting is what Brendan referred to as the 'core OS' - making Fedora run on ARM. Any ARM.
It just _happens_ that making the 'core OS' work gives you a reasonable chance at making, say, a basic web server fly than it does making a multitouch desktop operating system fly. So it kind of 'looks' right now like Fedora ARM is more for a web server than it is for a multitouch desktop OS. But that's kind of misleading. The fact is that the 'core OS' needs to be there for both, so working on the 'core OS' is really working on both use cases. It just happens to be the case that one of the two use cases looks pretty viable almost as soon as the 'core OS' is done, and the other doesn't. The project is still, in the end, working on both.
On Tue, 2012-03-20 at 18:08 +0100, Kevin Kofler wrote:
Jakub Jelinek wrote:
Looking at last gcc build times (not unusual, though I really remember arm taking much longer than that, e.g. 4.7.0-0.11.fc17 took almost 17 hours on both arm architectures), from http://*koji.fedoraproject.org/packages/gcc/4.7.0/0.20.fc17/data/logs/*/stat... : i686 4h18m x86_64 1h25m ppc 4h12m ppc64 4h16m s390 6h27m s390x 6h04m armv5tel 26h20m armv7hl 24h17m
So even speeding this up twice means it is still 2x slower than the slowest other secondary architecture.
Ouch!!!
That shows that ARM should be the LAST architecture we consider for primary status rather than the first. (That said, I don't think it makes sense to make PPC primary again or to make S/390 primary. They don't have anywhere near the market share. But IMHO ARM doesn't have the market share either.)
Can you define what market you refer to ?
Simo.
On 03/20/2012 09:15 AM, Jakub Jelinek wrote:
Looking at last gcc build times (not unusual, though I really remember arm taking much longer than that, e.g. 4.7.0-0.11.fc17 took almost 17 hours on both arm architectures), from http://*koji.fedoraproject.org/packages/gcc/4.7.0/0.20.fc17/data/logs/*/stat... : i686 4h18m x86_64 1h25m ppc 4h12m ppc64 4h16m s390 6h27m s390x 6h04m armv5tel 26h20m armv7hl 24h17m
So even speeding this up twice means it is still 2x slower than the slowest other secondary architecture.
Can Koji use distcc for ARM arches? Would that speedup be enough to make ARM build competitive with others?
-- Andy
On 03/20/2012 01:14 PM, Andy Grover wrote:
Can Koji use distcc for ARM arches? Would that speedup be enough to make ARM build competitive with others?
I believe this is a non-starter for rel-eng. The ARM team are not recommending this path.
On 03/20/2012 12:15 PM, Jakub Jelinek wrote:
Looking at last gcc build times (not unusual, though I really remember arm taking much longer than that, e.g. 4.7.0-0.11.fc17 took almost 17 hours on both arm architectures), from http://*koji.fedoraproject.org/packages/gcc/4.7.0/0.20.fc17/data/logs/*/stat... : i686 4h18m x86_64 1h25m
...
armv5tel 26h20m armv7hl 24h17m
Is cross-compile an option? if it is, how long does it take to cross-compile in an x86_64 environment?
On 03/20/2012 04:58 PM, Brendan Conoboy wrote:
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line? No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow?
My #1 problem would be "making packages compilable" and bug-hunting arch-specific compilation problems.
If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
Yes, because "building primary archs first" doesn't help when build-problems are secondary arch specific.
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
Ralf
On 03/20/2012 09:21 AM, Ralf Corsepius wrote:
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into. Let's figure out how to make native compilation work *better*, how to make koji work *better* when more architectures are involved than just x86.
On Tue, Mar 20, 2012 at 5:34 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:21 AM, Ralf Corsepius wrote:
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
Let's figure out how to make native compilation work *better*, how to make koji work *better* when more architectures are involved than just x86.
The hardware is way slower ... so we can just build on faster hardware (x86_64). Which is the only sane way to do it. Trying to build on ARM directly is kind of a gimmick but nothing one can seriously use to build a whole operating system. (Yes it works but it is way to slow).
On Tue, Mar 20, 2012 at 05:37:10PM +0100, drago01 wrote:
On Tue, Mar 20, 2012 at 5:34 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:21 AM, Ralf Corsepius wrote:
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
We use cross-compilation right now for mingw-* packages (for Windows). However you cannot use cross-compilation to create a foo-*.armv7hl.rpm package. That's because our entire toolchain, from RPM through Koji, simply does not understand cross-compilation properly.
Solvable, but undoubtedly a ton of work for everyone.
Rich.
On Tue, Mar 20, 2012 at 5:46 PM, Richard W.M. Jones rjones@redhat.com wrote:
On Tue, Mar 20, 2012 at 05:37:10PM +0100, drago01 wrote:
On Tue, Mar 20, 2012 at 5:34 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:21 AM, Ralf Corsepius wrote:
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
We use cross-compilation right now for mingw-* packages (for Windows). However you cannot use cross-compilation to create a foo-*.armv7hl.rpm package. That's because our entire toolchain, from RPM through Koji, simply does not understand cross-compilation properly.
Solvable, but undoubtedly a ton of work for everyone.
I don't know about the details there but that does not sound like unfixable to be. I'd even say that fixing that is a prerequisite to allow secondary archs that run on "slow" hardware to become primary.
On 03/20/2012 09:50 AM, drago01 wrote:
I don't know about the details there but that does not sound like unfixable to be. I'd even say that fixing that is a prerequisite to allow secondary archs that run on "slow" hardware to become primary.
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
On 03/20/2012 12:56 PM, Brendan Conoboy wrote:
On 03/20/2012 09:50 AM, drago01 wrote:
I don't know about the details there but that does not sound like unfixable to be. I'd even say that fixing that is a prerequisite to allow secondary archs that run on "slow" hardware to become primary.
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Yup. I vote we shelve this part of the discussion in the interest of ever turning our proposal into something that will be accepted.
Jon.
On Tue, Mar 20, 2012 at 5:57 PM, Jon Masters jcm@redhat.com wrote:
On 03/20/2012 12:56 PM, Brendan Conoboy wrote:
On 03/20/2012 09:50 AM, drago01 wrote:
I don't know about the details there but that does not sound like unfixable to be. I'd even say that fixing that is a prerequisite to allow secondary archs that run on "slow" hardware to become primary.
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Yup. I vote we shelve this part of the discussion in the interest of ever turning our proposal into something that will be accepted.
OK fair enough (even though I still think that it is the only sane way to solve the build speed issue).
Brendan Conoboy wrote:
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Possible. That just means ARM cannot become a primary architecture any time soon.
We should not ignore the problem just because you cannot solve it. It is a blocking issue.
Kevin Kofler
On Tue, Mar 20, 2012 at 5:56 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:50 AM, drago01 wrote:
I don't know about the details there but that does not sound like unfixable to be. I'd even say that fixing that is a prerequisite to allow secondary archs that run on "slow" hardware to become primary.
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Sorry I am not buying that.
drago01 wrote:
On Tue, Mar 20, 2012 at 5:56 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:50 AM, drago01 wrote:
I don't know about the details there but that does not sound like unfixable to be. I'd even say that fixing that is a prerequisite to allow secondary archs that run on "slow" hardware to become primary.
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Sorry I am not buying that.
I think you're underestimating the problems. RPM is not designed for cross- compilation, at all. Not all upstream build systems support it, either. And even where the build system supports it in principle, the individual upstream package might be doing something which actually makes it fail. While I'm not sure it would require decades, I also doubt it will be a workable solution in the short term.
I think the current secondary architecture setup is the much better solution, it lets ARM builders complete their builds at their own pace without slowing down everyone.
Kevin Kofler
On 03/20/2012 10:44 AM, drago01 wrote:
On Tue, Mar 20, 2012 at 5:56 PM, Brendan Conoboyblc@redhat.com wrote:
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Sorry I am not buying that.
Because you have vast experience to the contrary? Look, even x86_64 is topping out on speed and moving to a more-core and more-systems-per-rack model. Cross compilation solves yesterday's problem, not tomorrow's. If build speed truly is a fundamental issue to becoming PA the answer is to harness multiple systems for a single build, not to use a somewhat faster system to make up for the speed of a somewhat slower system. Scaling across more cores than fit in a single SMP Linux environment is the only sensible approach to future build speedups. Though is an interesting challenge, it is completely beyond the scope of primary architecture requirements. Please, let's drop talk of cross compilation.
On Tue, Mar 20, 2012 at 7:05 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 10:44 AM, drago01 wrote:
On Tue, Mar 20, 2012 at 5:56 PM, Brendan Conoboyblc@redhat.com wrote:
Please, please, no. Cross compilation for Fedora cannot and will not ever
get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Sorry I am not buying that.
Because you have vast experience to the contrary?
No I do believe that there is some work required to do it. But *man-decades* is just an overstatement.
Look, even x86_64 is topping out on speed and moving to a more-core and more-systems-per-rack model.
I didn't claim otherwise ...
Cross compilation solves yesterday's problem, not tomorrow's.
Not really no. Even if we get ARM up to reasonable speeds it could help other arches in the future.
If build speed truly is a fundamental issue to becoming PA the answer is to harness multiple systems for a single build, not to use a somewhat faster system to make up for the speed of a somewhat slower system.
Sure if you can solve the problem in a different way it is fine. I just said then when I did software development on ARM building the software on an x86_64 host was the only obvious choice. So it sounds logical to apply it here where the goal is to build a *whole operating system* and not just a specific program. (OK hardware advanced since then but still).
Scaling across more cores than fit in a single SMP Linux environment is the only sensible approach to future build speedups.
Fine I am not going to stop you from doing that.
Though is an interesting challenge, it is completely beyond the scope of primary architecture requirements. Please, let's drop talk of cross compilation.
As I said in the other mail I am fine with that. I just had to respond to the "man-decades" hyperbole. (Maybe I should have just ignored it).
On 03/20/2012 11:15 AM, drago01 wrote:
As I said in the other mail I am fine with that. I just had to respond to the "man-decades" hyperbole. (Maybe I should have just ignored it).
Okay, feel free to send me private mail or ask in IRC and we can discuss, if only for academic purposes :-)
On Tue, Mar 20, 2012 at 11:05:20AM -0700, Brendan Conoboy wrote:
On 03/20/2012 10:44 AM, drago01 wrote:
On Tue, Mar 20, 2012 at 5:56 PM, Brendan Conoboyblc@redhat.com wrote:
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Sorry I am not buying that.
Because you have vast experience to the contrary? Look, even x86_64 is topping out on speed and moving to a more-core and more-systems-per-rack model. Cross compilation solves yesterday's problem, not tomorrow's. If build speed truly is a fundamental issue to becoming PA the answer is to harness multiple systems for a single build, not to use a somewhat faster system to make up for the speed of a somewhat slower system. Scaling across more cores than fit in a single SMP Linux environment is the only sensible approach to future build speedups.
This is a sound observation, but it doesn't apply to Fedora on ARM unless you can demonstrate a system where rpmbuild scales well across multiple ARM cores/servers/whatever.
I would suggest -- in order to move the present discussion on -- that you try using various methods to speed up an ARM build of (eg) glibc. distcc, some sort of demo cross-compilation, etc. What works, what doesn't work, what needs more work?
Rich.
On 03/20/2012 01:48 PM, Richard W.M. Jones wrote:
I would suggest -- in order to move the present discussion on -- that you try using various methods to speed up an ARM build of (eg) glibc. distcc, some sort of demo cross-compilation, etc. What works, what doesn't work, what needs more work?
Distcc suffers from a fatal flaw: You can't reproduce the build. Transient failures and corruption cannot be tracked down. We've done lots of experimentation with distcc, even using distcc + a cross compiler. It does great stuff to build time performance, but it sacrifices build integrity for performance and that's not an acceptable condition for PA.
On 03/20/2012 07:05 PM, Brendan Conoboy wrote:
On 03/20/2012 10:44 AM, drago01 wrote:
On Tue, Mar 20, 2012 at 5:56 PM, Brendan Conoboyblc@redhat.com wrote:
Please, please, no. Cross compilation for Fedora cannot and will not ever get a secondary arch to primary. We're talking man-decades of engineering time to solve all the problems. Decades.
Sorry I am not buying that.
Because you have vast experience to the contrary? Look, even x86_64 is topping out on speed and moving to a more-core and more-systems-per-rack model. Cross compilation solves yesterday's problem, not tomorrow's.
Today's problem are:
1. sufficiently performant arms are not wide-spread and 2. today's code is hardly tested on arms.
IMO, wrt. Fedora, unless a solution to 1. can be provided to the Fedora developer community, 2. can not be achieved and arm-Fedora will remain an illusion.
If build speed truly is a fundamental issue to becoming PA the answer is to harness multiple systems for a single build, not to use a somewhat faster system to make up for the speed of a somewhat slower system. Scaling across more cores than fit in a single SMP Linux environment is the only sensible approach to future build speedups. Though is an interesting challenge, it is completely beyond the scope of primary architecture requirements. Please, let's drop talk of cross compilation.
I agree insofar as cross-compilation is a band-aid. However, sometimes, band-aids are inevitable.
Ralf
drago01 wrote:
I don't know about the details there but that does not sound like unfixable to be.
In theory yes, in practice I don't think this will be fixed any time soon, yet…
I'd even say that fixing that is a prerequisite to allow secondary archs that run on "slow" hardware to become primary.
… I have to wholeheartedly agree with you on this part. It's just not possible to wait for those glacially slow ARM builders to complete their builds. Even the "enterprise class" hardware being discussed is going to be ridiculously slow.
Kevin Kofler
On 03/20/2012 05:46 PM, Richard W.M. Jones wrote:
On Tue, Mar 20, 2012 at 05:37:10PM +0100, drago01 wrote:
On Tue, Mar 20, 2012 at 5:34 PM, Brendan Conoboyblc@redhat.com wrote:
On 03/20/2012 09:21 AM, Ralf Corsepius wrote:
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
We use cross-compilation right now for mingw-* packages (for Windows). However you cannot use cross-compilation to create a foo-*.armv7hl.rpm package. That's because our entire toolchain, from RPM through Koji, simply does not understand cross-compilation properly.
Well, the mock/rpm part is the smaller part of the issues (I use customized mock setups on Fedora to build mingw-* and cygwin-* packages).
Solvable, but undoubtedly a ton of work for everyone.
The real issue would be to re-utilize "foreign native rpms" (here *.arm.rpms) to install them in sys-roots on x86.
(Fedora's mingw*-toolchains are explictly packaged to fit into x86)
Ralf
On Tuesday, March 20, 2012, 7:21:25 PM, Ralf Corsepius wrote:
On 03/20/2012 05:46 PM, Richard W.M. Jones wrote:
On Tue, Mar 20, 2012 at 05:37:10PM +0100, drago01 wrote:
On Tue, Mar 20, 2012 at 5:34 PM, Brendan Conoboyblc@redhat.com wrote:
On 03/20/2012 09:21 AM, Ralf Corsepius wrote:
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
We use cross-compilation right now for mingw-* packages (for Windows). However you cannot use cross-compilation to create a foo-*.armv7hl.rpm package. That's because our entire toolchain, from RPM through Koji, simply does not understand cross-compilation properly.
Well, the mock/rpm part is the smaller part of the issues (I use customized mock setups on Fedora to build mingw-* and cygwin-* packages).
Solvable, but undoubtedly a ton of work for everyone.
The real issue would be to re-utilize "foreign native rpms" (here *.arm.rpms) to install them in sys-roots on x86.
(Fedora's mingw*-toolchains are explictly packaged to fit into x86) Ralf
On July 7th, 2009, Mark Salter made a post "crossbuilding rpms with koji" on the fedora-buildsys-list" where he described a project to add cross-building support to koji/moc/rpm/etc.
The post in archived post is at http://www.redhat.com/archives/fedora-buildsys-list/2009-July/msg00000.html
On 03/20/2012 09:37 AM, drago01 wrote:
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
Okay, why not?
The ones off the top of my head, and this is by no means exhaustive:
1. Fedora Policy (Which I imagine is based on the technical foundation of the following 5+ points and others I'm unaware of).
2. Many packages assume a native execution environment which will not exist. Incredible undertaking to move 11000 packages to cross compilation framework.
3. Absence of arm-linux cross compilers in the distribution.
4. If there were arm-linux cross compilers, how do you keep them in sync with native gcc?
5. Where does the sys-root for an arm-linux cross compiler come from?
6. Would koji then be native/cross ambidextrous? Who is going to do that?
For all these reasons and more we're not proposing cross compilation for ARM. Just doing so defies what it means to be PA.
The hardware is way slower ... so we can just build on faster hardware (x86_64). Which is the only sane way to do it. Trying to build on ARM directly is kind of a gimmick but nothing one can seriously use to build a whole operating system. (Yes it works but it is way to slow).
In couple years the hardware is going to be surprisingly comparable or exceed to what you're see on x86, especially as the number of cores skyrockets while the GHz continue to climb. It's not a gimmick, we're just preparing for the future before it gets here. The only problem we face is that those cores are in multiple CPUs so we can't 'make -j' our way out of the build system problem.
On Tue, Mar 20, 2012 at 5:50 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:37 AM, drago01 wrote:
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
Okay, why not?
The ones off the top of my head, and this is by no means exhaustive:
- Fedora Policy (Which I imagine is based on the technical foundation of
the following 5+ points and others I'm unaware of).
I said "technical" so lets take policy aside ...
- Many packages assume a native execution environment which will not exist.
Incredible undertaking to move 11000 packages to cross compilation framework.
qemu? Should be still faster then doing the whole build on arm.
- Absence of arm-linux cross compilers in the distribution.
Err yeah but nothing that can't be fixed.
- If there were arm-linux cross compilers, how do you keep them in sync
with native gcc?
Build from the same srpm.
- Where does the sys-root for an arm-linux cross compiler come from?
- Would koji then be native/cross ambidextrous? Who is going to do that?
No real answers to them yet but fixing them seems to be easier then "make arm as fast as x86_64".
For all these reasons and more we're not proposing cross compilation for ARM. Just doing so defies what it means to be PA.
We should somehow define what a PA is then. I wouldn't have added "built on native hardware" because that does not really seem to matter.
The hardware is way slower ... so we can just build on faster hardware (x86_64). Which is the only sane way to do it. Trying to build on ARM directly is kind of a gimmick but nothing one can seriously use to build a whole operating system. (Yes it works but it is way to slow).
In couple years the hardware is going to be surprisingly comparable or exceed to what you're see on x86, especially as the number of cores skyrockets while the GHz continue to climb.
Might be true might be not ... we are talking about the next couple of months not years.
It's not a gimmick, we're just preparing for the future before it gets here. The only problem we face is that those cores are in multiple CPUs so we can't 'make -j' our way out of the build system problem.
Huh? Not sure I follow here.
Also I am not opposed to having ARM as PA, I just don't really think we should do it the way it is proposed here (build on hardware that is way slower then the rest of the builders).
drago01 píše v Út 20. 03. 2012 v 17:57 +0100:
On Tue, Mar 20, 2012 at 5:50 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:37 AM, drago01 wrote:
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
Okay, why not?
The ones off the top of my head, and this is by no means exhaustive:
- Fedora Policy (Which I imagine is based on the technical foundation of
the following 5+ points and others I'm unaware of).
I said "technical" so lets take policy aside ...
- Many packages assume a native execution environment which will not exist.
Incredible undertaking to move 11000 packages to cross compilation framework.
qemu? Should be still faster then doing the whole build on arm.
just a side note - I was told by an OpenSUSE on ARM person that they use x86 boxes with the user-space qemu virtual machine. It works quite fast, but still needs some hacking eg. in test-suites
Dan
On Tue, 2012-03-20 at 18:21 +0100, Dan Horák wrote:
drago01 píše v Út 20. 03. 2012 v 17:57 +0100:
On Tue, Mar 20, 2012 at 5:50 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:37 AM, drago01 wrote:
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
Okay, why not?
The ones off the top of my head, and this is by no means exhaustive:
- Fedora Policy (Which I imagine is based on the technical foundation of
the following 5+ points and others I'm unaware of).
I said "technical" so lets take policy aside ...
- Many packages assume a native execution environment which will not exist.
Incredible undertaking to move 11000 packages to cross compilation framework.
qemu? Should be still faster then doing the whole build on arm.
just a side note - I was told by an OpenSUSE on ARM person that they use x86 boxes with the user-space qemu virtual machine. It works quite fast, but still needs some hacking eg. in test-suites
With Qemu, my $1200 i7 quadcore desktop can successfully emulate a $129 GuruPlug. It just doesn't make a lot of sense to go this route.
-Chris
Chris Tyler píše v Út 20. 03. 2012 v 14:40 -0400:
On Tue, 2012-03-20 at 18:21 +0100, Dan Horák wrote:
drago01 píše v Út 20. 03. 2012 v 17:57 +0100:
On Tue, Mar 20, 2012 at 5:50 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:37 AM, drago01 wrote:
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
Okay, why not?
The ones off the top of my head, and this is by no means exhaustive:
- Fedora Policy (Which I imagine is based on the technical foundation of
the following 5+ points and others I'm unaware of).
I said "technical" so lets take policy aside ...
- Many packages assume a native execution environment which will not exist.
Incredible undertaking to move 11000 packages to cross compilation framework.
qemu? Should be still faster then doing the whole build on arm.
just a side note - I was told by an OpenSUSE on ARM person that they use x86 boxes with the user-space qemu virtual machine. It works quite fast, but still needs some hacking eg. in test-suites
With Qemu, my $1200 i7 quadcore desktop can successfully emulate a $129 GuruPlug. It just doesn't make a lot of sense to go this route.
no, I don't want to recommend it, but I'd expect that the user-space (syscall) emulation in qemu would be faster than the full system emulation, because all I/O is native speed.
Dan
----- Original Message -----
just a side note - I was told by an OpenSUSE on ARM person that they use x86 boxes with the user-space qemu virtual machine. It works quite fast, but still needs some hacking eg. in test-suites
Yep, OpenSUSE uses qemu - it's sometimes not as stable as it should be, I saw a few strange random crashes in qemu but usually it works. I talked with them once, they were surprised by our "native builds" policy.
Maybe it's worth to ask them (or look at for example Mer builds) what's the difference in build times.
R.
Dan
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
----- Original Message -----
Maybe it's worth to ask them (or look at for example Mer builds) what's the difference in build times.
A few statistics from build.meego.com - using the OBS and building in qemu. These are really just approximate numbers, built in different times with probably a different load...
I took Qt as an example as it's a package I know.
------ build.meego.com ------- http://build.meego.com/package/show?package=qt&project=Trunk armv8el build19 started "build qt.spec" at Sat Nov 5 02:09:33 UTC 2011. build19 finished "build qt.spec" at Sat Nov 5 03:01:43 UTC 2011.
approx. 1 hour
i586 build17 started "build qt.spec" at Fri Nov 4 23:33:24 UTC 2011. build17 finished "build qt.spec" at Sat Nov 5 00:05:03 UTC 2011.
approx. half hour (1/2)
armv8el vs i586 factor of 2
http://build.meego.com/package/show?package=qt&project=home%3Arrojfors%3... armv7el build42 started "build qt.spec" at Thu May 12 08:49:50 UTC 2011. build42 finished "build qt.spec" at Thu May 12 10:42:21 UTC 2011.
approx. 2 hours
i586 build11 started "build qt.spec" at Thu May 12 08:49:48 UTC 2011. build11 finished "build qt.spec" at Thu May 12 09:09:47 UTC 2011.
approx.
armv7el vs i586 factor of 4
------ Fedora ------ i686 2012-02-20 14:31:51,510 - Mock Version: 1.1.18 2012-02-20 15:05:21,089 - State Changed: end
approx. half hour
armv7hl 2012-03-18 17:58:09,566 - Mock Version: 1.1.18 2012-03-19 04:53:07,593 - State Changed: end
better not calculating...
So probably using Qemu could speed it up quite a lot. Also OBS offers quite a lot of flexibility to decouple arch builds, disable selected archs etc. But I'm not sure about the processes for chain builds, updates, how they make the builds consistent (if one arch fails)...
From package POV:
* the KDE stack would suffer a lot of slow builds due to dependencies, even now it can take half day to have all bootstrapped * currently we omit all ARM patches available in upstreams as we don't need them and we don't have a chance to test even the patch is needed :( * for fixing bugs, we could always try to reach any ARM guy to help with
Maybe a distribution of PandaBoards/R-Pi for every FAS account holder could help, any sponsor? :D
/me is going to order R-pi soon :)
R.
R.
Dan
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Wed, Mar 21, 2012 at 11:07 AM, Jaroslav Reznik jreznik@redhat.com wrote:
----- Original Message -----
Maybe it's worth to ask them (or look at for example Mer builds) what's the difference in build times.
A few statistics from build.meego.com - using the OBS and building in qemu. These are really just approximate numbers, built in different times with probably a different load...
I took Qt as an example as it's a package I know.
------ build.meego.com ------- http://build.meego.com/package/show?package=qt&project=Trunk armv8el build19 started "build qt.spec" at Sat Nov 5 02:09:33 UTC 2011. build19 finished "build qt.spec" at Sat Nov 5 03:01:43 UTC 2011.
approx. 1 hour
i586 build17 started "build qt.spec" at Fri Nov 4 23:33:24 UTC 2011. build17 finished "build qt.spec" at Sat Nov 5 00:05:03 UTC 2011.
approx. half hour (1/2)
armv8el vs i586 factor of 2
http://build.meego.com/package/show?package=qt&project=home%3Arrojfors%3... armv7el build42 started "build qt.spec" at Thu May 12 08:49:50 UTC 2011. build42 finished "build qt.spec" at Thu May 12 10:42:21 UTC 2011.
approx. 2 hours
i586 build11 started "build qt.spec" at Thu May 12 08:49:48 UTC 2011. build11 finished "build qt.spec" at Thu May 12 09:09:47 UTC 2011.
approx.
armv7el vs i586 factor of 4
------ Fedora ------ i686 2012-02-20 14:31:51,510 - Mock Version: 1.1.18 2012-02-20 15:05:21,089 - State Changed: end
approx. half hour
armv7hl 2012-03-18 17:58:09,566 - Mock Version: 1.1.18 2012-03-19 04:53:07,593 - State Changed: end
better not calculating...
So probably using Qemu could speed it up quite a lot. Also OBS offers
Those numbers look way better then Kevin's "50x slower without any citation" ... thanks for getting this numbers.
drago01 wrote:
Those numbers look way better then Kevin's "50x slower without any citation" ... thanks for getting this numbers.
I'm surprised emulating ARM in QEMU is so much faster than qemu-system- x86_64 (which was how I measured the 50 times). Are they really using QEMU for everything or are they actually using host or cross tools (and QEMU only for when the build process is trying to run a just built binary)?
That said, a factor of 2 to 4 is still too slow!
Kevin Kofler
On Thu, Mar 22, 2012 at 1:56 AM, Kevin Kofler kevin.kofler@chello.at wrote:
drago01 wrote:
Those numbers look way better then Kevin's "50x slower without any citation" ... thanks for getting this numbers.
I'm surprised emulating ARM in QEMU is so much faster than qemu-system- x86_64 (which was how I measured the 50 times).
Given that x86_64 is way more complex then ARM it is not *that* surprising.
Are they really using QEMU for everything or are they actually using host or cross tools (and QEMU only for when the build process is trying to run a just built binary)?
The later i.e cross tools + user emulation for binaries.
On Thu, Mar 22, 2012 at 8:32 AM, drago01 drago01@gmail.com wrote:
On Thu, Mar 22, 2012 at 1:56 AM, Kevin Kofler kevin.kofler@chello.at wrote:
drago01 wrote:
Those numbers look way better then Kevin's "50x slower without any citation" ... thanks for getting this numbers.
I'm surprised emulating ARM in QEMU is so much faster than qemu-system- x86_64 (which was how I measured the 50 times).
Given that x86_64 is way more complex then ARM it is not *that* surprising.
Are they really using QEMU for everything or are they actually using host or cross tools (and QEMU only for when the build process is trying to run a just built binary)?
The later i.e cross tools + user emulation for binaries.
OK as Dennis said it seems they also run the compilers through qemu.
drago01 wrote:
On Thu, Mar 22, 2012 at 1:56 AM, Kevin Kofler kevin.kofler@chello.at wrote:
I'm surprised emulating ARM in QEMU is so much faster than qemu-system- x86_64 (which was how I measured the 50 times).
Given that x86_64 is way more complex then ARM it is not *that* surprising.
I think the fact that they're using userspace emulation rather than system emulation is actually the biggest difference to the setup I was using, and accounts for a significant part of the difference in slowdown. (Note that userspace emulation wasn't an option at all in my setup, for x86_64 on 32- bit x86.) To know for sure, we'd have to measure how long it takes to do builds with qemu-system-arm, assuming we can get that to work at all.
Kevin Kofler
On Wed, Mar 21, 2012 at 10:07 AM, Jaroslav Reznik jreznik@redhat.com wrote:
----- Original Message -----
Maybe it's worth to ask them (or look at for example Mer builds) what's the difference in build times.
A few statistics from build.meego.com - using the OBS and building in qemu. These are really just approximate numbers, built in different times with probably a different load...
I took Qt as an example as it's a package I know.
------ build.meego.com ------- http://build.meego.com/package/show?package=qt&project=Trunk armv8el build19 started "build qt.spec" at Sat Nov 5 02:09:33 UTC 2011. build19 finished "build qt.spec" at Sat Nov 5 03:01:43 UTC 2011.
approx. 1 hour
i586 build17 started "build qt.spec" at Fri Nov 4 23:33:24 UTC 2011. build17 finished "build qt.spec" at Sat Nov 5 00:05:03 UTC 2011.
approx. half hour (1/2)
armv8el vs i586 factor of 2
http://build.meego.com/package/show?package=qt&project=home%3Arrojfors%3... armv7el build42 started "build qt.spec" at Thu May 12 08:49:50 UTC 2011. build42 finished "build qt.spec" at Thu May 12 10:42:21 UTC 2011.
approx. 2 hours
i586 build11 started "build qt.spec" at Thu May 12 08:49:48 UTC 2011. build11 finished "build qt.spec" at Thu May 12 09:09:47 UTC 2011.
approx.
armv7el vs i586 factor of 4
------ Fedora ------ i686 2012-02-20 14:31:51,510 - Mock Version: 1.1.18 2012-02-20 15:05:21,089 - State Changed: end
approx. half hour
armv7hl 2012-03-18 17:58:09,566 - Mock Version: 1.1.18 2012-03-19 04:53:07,593 - State Changed: end
better not calculating...
So probably using Qemu could speed it up quite a lot. Also OBS offers quite a lot of flexibility to decouple arch builds, disable selected archs etc. But I'm not sure about the processes for chain builds, updates, how they make the builds consistent (if one arch fails)...
All sorts of things can speed it up, most of the Fedora builders are currently loopback ext4 over NFS over 100Mb ethernet over USB. Not optimal. Add to that 1Gb of RAM and swap the problem gets worse. The devices we're looking at have proper SATA ports (not over USB) and quad core 4GB RAM and the time to build is an order of magnitude faster, and those boards aren't overly stable as they're not production level HW so once we get our hands on production level versions of that HW we can start to properly test the difference in large packages such as gcc, qt, libreoffice and the kernel and will be able to much better ascertain the impact. I believe that should be "soon" although I'm not in direct contact.
Peter
On Wed, Mar 21, 2012 at 01:27:04PM +0000, Peter Robinson wrote:
All sorts of things can speed it up, most of the Fedora builders are currently loopback ext4 over NFS over 100Mb ethernet over USB. Not optimal.
Just switching them to ext2 would save a ton of IO. The buildroots get regenerated every time anyway, so journalling isn't really getting you anything. If a builder crashes, you probably want to throw the old buildroot away and start over anyway.
out of curiousity, Is the setup of the builders documented somewhere ?
Dave
On Wed, Mar 21, 2012 at 2:58 PM, Dave Jones davej@redhat.com wrote:
On Wed, Mar 21, 2012 at 01:27:04PM +0000, Peter Robinson wrote: > All sorts of things can speed it up, most of the Fedora builders are > currently loopback ext4 over NFS over 100Mb ethernet over USB. Not > optimal.
Just switching them to ext2 would save a ton of IO. The buildroots get regenerated every time anyway, so journalling isn't really getting you anything. If a builder crashes, you probably want to throw the old buildroot away and start over anyway.
out of curiousity, Is the setup of the builders documented somewhere ?
I believe it is somewhere, the reference I had is completely out of date. The current configuration would not be the configuration used in primary arch but ultimately it was the best we could do with the platforms that were available 12-18 months ago. A lot has changed in that time.
Peter
On 03/21/2012 10:58 AM, Dave Jones wrote:
On Wed, Mar 21, 2012 at 01:27:04PM +0000, Peter Robinson wrote:
All sorts of things can speed it up, most of the Fedora builders are currently loopback ext4 over NFS over 100Mb ethernet over USB. Not optimal.
Just switching them to ext2 would save a ton of IO.
You could also turn off the journal in ext4. That'd lose the journal IO and stalls waiting for it and you'd still get the block mapping IO savings of delalloc and extents.
- z
On Wed, Mar 21, 2012 at 11:32:04AM -0400, Zach Brown wrote:
On 03/21/2012 10:58 AM, Dave Jones wrote:
On Wed, Mar 21, 2012 at 01:27:04PM +0000, Peter Robinson wrote:
All sorts of things can speed it up, most of the Fedora builders are currently loopback ext4 over NFS over 100Mb ethernet over USB. Not optimal.
Just switching them to ext2 would save a ton of IO.
You could also turn off the journal in ext4. That'd lose the journal IO and stalls waiting for it and you'd still get the block mapping IO savings of delalloc and extents.
Yes, that would be even better. (Hi Zab!)
Dave
So probably using Qemu could speed it up quite a lot. Also OBS offers quite a lot of flexibility to decouple arch builds, disable selected archs etc. But I'm not sure about the processes for chain builds, updates, how they make the builds consistent (if one arch fails)...
All sorts of things can speed it up, most of the Fedora builders are currently loopback ext4 over NFS over 100Mb ethernet over USB. Not optimal. Add to that 1Gb of RAM and swap the problem gets worse. The devices we're looking at have proper SATA ports (not over USB) and quad core 4GB RAM and the time to build is an order of magnitude faster, and those boards aren't overly stable as they're not production level HW so once we get our hands on production level versions of that HW we can start to properly test the difference in large packages such as gcc, qt, libreoffice and the kernel and will be able to much better ascertain the impact. I believe that should be "soon" although I'm not in direct contact.
It was more argument about real qemu speed from real deployment. Not bashing the current setup.
It's better to have real data over just talking here :)
R.
Peter
devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 21 Mar 2012 06:07:57 -0400 (EDT) Jaroslav Reznik jreznik@redhat.com wrote:
----- Original Message -----
Maybe it's worth to ask them (or look at for example Mer builds) what's the difference in build times.
A few statistics from build.meego.com - using the OBS and building in qemu. These are really just approximate numbers, built in different times with probably a different load...
I have spoken with the OpenSUSE guys they dont use qemu-system-arm but rather qemu-arm and lay out things and build using a hybrid environment thats also how they build ppc s390 and other arches. the only build hardware they have is x86. doing full system emulation will be slower.
Dennis
On Wed, Mar 21, 2012 at 7:11 PM, Dennis Gilmore dennis@ausil.us wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 21 Mar 2012 06:07:57 -0400 (EDT) Jaroslav Reznik jreznik@redhat.com wrote:
----- Original Message -----
Maybe it's worth to ask them (or look at for example Mer builds) what's the difference in build times.
A few statistics from build.meego.com - using the OBS and building in qemu. These are really just approximate numbers, built in different times with probably a different load...
I have spoken with the OpenSUSE guys they dont use qemu-system-arm but rather qemu-arm and lay out things and build using a hybrid environment thats also how they build ppc s390 and other arches. the only build hardware they have is x86. doing full system emulation will be slower.
Which is exactly what I proposed ... i.e use cross compilers and really on qemu to run stuff that gets generated and run during build.
But there seems to be a huge oppositions against that in Fedora. How does Ubuntu build there ARM builds? Native or using cross compilers?
On Wed, Mar 21, 2012 at 6:59 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/21/2012 11:18 AM, drago01 wrote:
But there seems to be a huge oppositions against that in Fedora. How does Ubuntu build there ARM builds? Native or using cross compilers?
Native.
As do Debian I believe. I think, but aren't 100% sure, that all major distributions except suse build as native.
Peter
On 03/21/2012 02:13 PM, Peter Robinson wrote:
As do Debian I believe. I think, but aren't 100% sure, that all major distributions except suse build as native.
At the last Linaro Connect the Debian guys said they're building natively on i.MX53 boards (Which are cool because they have real SATA).
Actually debian hfp is built on efika smart tops. They have a 8gb ssd attached using pata and 512mb ram.
On Wed, Mar 21, 2012 at 7:59 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/21/2012 11:18 AM, drago01 wrote:
But there seems to be a huge oppositions against that in Fedora. How does Ubuntu build there ARM builds? Native or using cross compilers?
Native.
OK kind of unexpected though.
On Wed, Mar 21, 2012 at 01:11:27PM -0500, Dennis Gilmore wrote:
I have spoken with the OpenSUSE guys they dont use qemu-system-arm but rather qemu-arm and lay out things and build using a hybrid environment thats also how they build ppc s390 and other arches. the only build hardware they have is x86. doing full system emulation will be slower.
In case it's not clear to the peanut gallery, qemu-system-arm and qemu-arm work in different ways.
qemu-system-arm emulates a complete machine, so your ARM kernel is also running and being emulated. qemu-arm just emulates the userspace part of the program, but translates system calls from the program and runs them against the regular (x86-64 in this case) kernel.
qemu-arm should be a lot faster, since the kernel part is running at full speed. On the other hand, there's some start-up overhead for every process run this way.
I should also say that in both cases qemu's "TCG" emulation is reasonably smart, and recompiles guest ARM code on the fly down to native (x86-64 in this case) code. Recommended reading: http://lugatgt.org/content/qemu_internals/downloads/slides.pdf
Rich.
On Wed, Mar 21, 2012 at 6:07 AM, Jaroslav Reznik jreznik@redhat.com wrote:
Maybe a distribution of PandaBoards/R-Pi for every FAS account holder could help, any sponsor? :D
OLPC is starting mass production of XO-1.75 units, based on an ARMv7 Marvell Armada 610. School kids in Uruguay and Nicaragua will start their school year with them, using a F14 ARM build. Peter Robinson has been instrumental in making this happen (and of course thanks to all the ARM porters).
We have free units for Fedora developers that are interested -- we'll ship them for free anywhere in the world. The numbers are limited (we are a non-profit), apply for units here:
http://wiki.laptop.org/go/Contributors_program#FAQ
Some additional discussion: http://lists.fedoraproject.org/pipermail/arm/2011-July/001661.html
In the form, you can state that your project is porting Fedora to ARMv7 ensuring it runs great on XO-1.75 :-) and mention what packages you're working on, or if you are or intend to be part of the Fedora ARM porters team, state so. We're on the fedora-arm mailing list, we'll know you :-)
F17 upgrade is coming midyear, which will give these units a big kick in the pants in terms of performance. After that, we are cooking some bold sw+hw plans for the future. All based on ARM.
cheers,
m
On Wed, 2012-03-21 at 05:04 -0400, Jaroslav Reznik wrote:
----- Original Message -----
just a side note - I was told by an OpenSUSE on ARM person that they use x86 boxes with the user-space qemu virtual machine. It works quite fast, but still needs some hacking eg. in test-suites
Yep, OpenSUSE uses qemu - it's sometimes not as stable as it should be, I saw a few strange random crashes in qemu but usually it works. I talked with them once, they were surprised by our "native builds" policy.
Maybe it's worth to ask them (or look at for example Mer builds) what's the difference in build times.
Fully-emulated actually fits into the "Native Builds" guideline, but it hasn't been economical to use this approach because there's no hardware support for ARM emulation on x86 (the way that there is hardware acceleration for x86 virtualization on x86) and it therefore requires a very fast/expensive x86 box to emulate a slow/cheap ARM box.
-Chris
On 03/21/2012 05:25 AM, Chris Tyler wrote:
Fully-emulated actually fits into the "Native Builds" guideline, but it hasn't been economical to use this approach because there's no hardware support for ARM emulation on x86 (the way that there is hardware acceleration for x86 virtualization on x86) and it therefore requires a very fast/expensive x86 box to emulate a slow/cheap ARM box.
The main place I see ARM emulation being useful is in allowing any packager with an x86 host to boot a simulated ARM host to resolve build failures in their package. That's not ideal- ideal is every package owner has an ARM system they can use, but it's perhaps a starting point. If other people have feedback on this point I'd be interested to hear their take on it. I think a combination of ARM emulation and providing a network-accessible set of test machines will go along way toward giving packagers what they need and plan to update the proposal to say so, subject to the feedback we get on this point.
On 3/21/12 10:36 AM, Brendan Conoboy wrote:
The main place I see ARM emulation being useful is in allowing any packager with an x86 host to boot a simulated ARM host to resolve build failures in their package. That's not ideal- ideal is every package owner has an ARM system they can use, but it's perhaps a starting point. If other people have feedback on this point I'd be interested to hear their take on it. I think a combination of ARM emulation and providing a network-accessible set of test machines will go along way toward giving packagers what they need and plan to update the proposal to say so, subject to the feedback we get on this point.
Arm emulation would go a long way toward validating produced install images too. Those of us that validate x86 images depend heavily on KVM and the like.
Jesse Keating wrote:
Arm emulation would go a long way toward validating produced install images too. Those of us that validate x86 images depend heavily on KVM and the like.
But full system emulation is slower by a LARGE factor, not merely the 2 to 4 Jaroslav quoted for OBS, which (according to Dennis) is using a hybrid setup including some host and cross tools and QEMU userspace emulation as a fallback, NOT full system emulation.
Kevin Kofler
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thu, 22 Mar 2012 02:02:59 +0100 Kevin Kofler kevin.kofler@chello.at wrote:
Jesse Keating wrote:
Arm emulation would go a long way toward validating produced install images too. Those of us that validate x86 images depend heavily on KVM and the like.
But full system emulation is slower by a LARGE factor, not merely the 2 to 4 Jaroslav quoted for OBS, which (according to Dennis) is using a hybrid setup including some host and cross tools and QEMU userspace emulation as a fallback, NOT full system emulation.
Kevin Kofler
its not using cross tools at all and i never said it is. They use qemu-arm to run the processes they run all arm binaries. rather than emulating the full system they run the processes emulated.
Dennis
On Tue, Mar 20, 2012 at 06:29:13PM +0100, Kevin Kofler wrote:
drago01 wrote:
qemu? Should be still faster then doing the whole build on arm.
LOL, no!
qemu software emulation slows down by a factor of ~50! Right now ARM is slower "only" by a factor of ~12.
Meh, at least you got something to _boot_ in qemu-system-arm.
Merely getting it to work at all has eluded me for years :-)
Rich.
Richard W.M. Jones wrote:
Meh, at least you got something to _boot_ in qemu-system-arm.
Actually, I haven't tried qemu-system-arm. The ~50 factor I quoted comes from my past experiences running qemu-system-x86_64 on a 32-bit machine to build x86_64 RPMs (before I got the Core 2 Duo notebook).
Kevin Kofler
Brendan Conoboy wrote:
In couple years the hardware is going to be surprisingly comparable or exceed to what you're see on x86, especially as the number of cores skyrockets while the GHz continue to climb.
Then let's rediscuss making ARM a primary architecture when that happens. Right now the speed is just not acceptable.
It's not a gimmick, we're just preparing for the future before it gets here. The only problem we face is that those cores are in multiple CPUs so we can't 'make -j' our way out of the build system problem.
That alone means it's not a solution.
Also note that not everything in builds is parallelizable: * not all upstream build systems use make -j, * configure (or cmake etc.) and make install are usually not parallelized, * often there are makefile dependencies requiring at least some amount of serialization etc. So even if you have -j working, you STILL need a comparable speed in ONE core compared to the x86 builders.
Kevin Kofler
On 03/20/2012 10:27 AM, Kevin Kofler wrote:
Then let's rediscuss making ARM a primary architecture when that happens. Right now the speed is just not acceptable.
Really? You're going to base your entire opinion on build time data on inappropriate hardware for one package without knowing even what the factors are in the build time? What if 50% of that time was due to test timeouts that require a simple fix? Please turn down the hyperbole dial and think before jumping to conclusions. If all Fedora is to you is a means of turning source code into binaries at lightning speed, x86 is great for you. I'm pretty sure Fedora is more than that. And if Fedora is going to be relevant in a few years time, it better support the CPU that is inside the computers everybody is buying today.
That alone means it's not a solution.
Also note that not everything in builds is parallelizable:
- not all upstream build systems use make -j,
But most of the big packages do.
- configure (or cmake etc.) and make install are usually not parallelized,
Make install speed is going to be the same. There will be no appreciable difference IO difference. It's entirely storage-speed bound.
- often there are makefile dependencies requiring at least some amount of serialization
Uh huh...
etc. So even if you have -j working, you STILL need a comparable speed in ONE core compared to the x86 builders.
Er, no, that's what you believe you need. I need packages to build in a period of time that works for the affected developers. You're responsible for some packages I believe so why don't you start by looking at how you would personally would be affected and talk about that? With as few exclamation points as possible, please. What's your slowest building package? We can probably extrapolate how long it will take to build with first generation ARM server hardware. From there we can talk about how that affects you work flow as well as how to handle it being delayed. Thanks,
Brendan Conoboy wrote:
Really? You're going to base your entire opinion on build time data on inappropriate hardware for one package without knowing even what the factors are in the build time? What if 50% of that time was due to test timeouts that require a simple fix? Please turn down the hyperbole dial and think before jumping to conclusions.
Those numbers were just one example, which reflects the trend I've experienced (also following the efforts to build KDE packages on the secondary architecture infrastructure) fairly well.
And in addition, build time is just one of the reasons (though the main one) why I don't think ARM should be a primary architecture.
If all Fedora is to you is a means of turning source code into binaries at lightning speed, x86 is great for you.
Indeed, x86 is great for fast builds, which are an important part of Fedora, because they're part of what allows us to deliver current software quickly and efficiently (e.g. we're usually the first binary distribution to offer the latest KDE Software Compilation bugfix release as an official, tested update – don't forget that "First" is part of our 4 'F's).
I'm pretty sure Fedora is more than that.
Sure, but that doesn't mean build times are not important.
And if Fedora is going to be relevant in a few years time, it better support the CPU that is inside the computers everybody is buying today.
1. It supports it. As a secondary architecture. Why is a secondary architecture not good enough support? (And why can't those issues be fixed, benefitting all secondary architectures?) 2. We can discuss making it a primary architecture "in a few years time", if ARM is really that successful by then. Why now? There's nothing requiring the change to be made NOW. 3. Your definition of "computers" is not the same as mine.
Er, no, that's what you believe you need. I need packages to build in a period of time that works for the affected developers. You're responsible for some packages I believe so why don't you start by looking at how you would personally would be affected and talk about that? With as few exclamation points as possible, please. What's your slowest building package?
Probably kdelibs.
We can probably extrapolate how long it will take to build with first generation ARM server hardware. From there we can talk about how that affects you work flow as well as how to handle it being delayed.
kdelibs is the first package which needs to be built for KDE SC updates and we cannot start building any of the others before it is complete. (And also note that there are some other interdependencies, e.g. several modules also depend on kde-workspace, which also takes a while to build.) Last I checked, Rex Dieter reported that kdelibs took many hours to build on the secondary architecture ARM builders even with apidocs disabled; with apidocs enabled (which is how we build it on the primary architectures; also note that the apidocs build process is NOT parallel), it was taking so long that it timed out entirely.
And in addition to individual packages taking too long to build, I also expect the aggregate latency due to many small package builds all taking 5 to 10 times as long to waste a lot of my time. I already find it painful to wait for my builds to complete even now!
Kevin Kofler
On Tue, 2012-03-20 at 09:50 -0700, Brendan Conoboy wrote:
- Fedora Policy (Which I imagine is based on the technical foundation
of the following 5+ points and others I'm unaware of).
- Many packages assume a native execution environment which will not
exist. Incredible undertaking to move 11000 packages to cross compilation framework.
And even if everything builds with cross tools, ee'd still need a native platform (hw or simulated) to run the %check stage.
Absence of arm-linux cross compilers in the distribution.
If there were arm-linux cross compilers, how do you keep them in sync
with native gcc?
Already some work being done on that front:
https://bugzilla.redhat.com/show_bug.cgi?id=761619 https://bugzilla.redhat.com/show_bug.cgi?id=766166
- Where does the sys-root for an arm-linux cross compiler come from?
The sys-root would be populated from already built RPMs much like the mock chroot.
Hello,
On 03/20/2012 12:37 PM, drago01 wrote:
On Tue, Mar 20, 2012 at 5:34 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:21 AM, Ralf Corsepius wrote:
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
Fedora generally doesn't cross-compile because you have to minimally run certain target configuration stuff on the host, and there are many other hardcoded expectations.
[ Aside - skip this bit - because someone is going to mention it and take this thread onto a wild tangent, yes you can use distcc hacks, yes, there is/was Scratchbox, and yes there are many other cute hacks. We haven't proposed any of this because we want to be boring, we want to win acceptance by doing what x86 does in as many cases as reasonable. It isn't reasonable to expect ARM to install using Anaconda on a $25 target, but it is reasonable to expect on-target build ].
Let's figure out how to make native compilation work *better*, how to make koji work *better* when more architectures are involved than just x86.
The hardware is way slower ... so we can just build on faster hardware (x86_64). Which is the only sane way to do it. Trying to build on ARM directly is kind of a gimmick but nothing one can seriously use to build a whole operating system. (Yes it works but it is way to slow).
Well, we've done a number of mass rebuilds, a complete bootstrap from scratch, and several releases now. So, it might be a "gimmick", but it works. We need to stop thinking of ARM as it was 10 years ago. This year, we're going to see systems with 288+ cores in 2U of rack space. Next year, we're going to see Cortex A-15 systems that will be much faster still, and the year after, we're going to see 64-bit systems with at least 8 highly performing cores. It's not all about performance though. ARM isn't going to beat x86 in a speed race...that is not the goal. It's about aggregate performance, not individual node performance at the high end, and about mass availability at the low end.
We can remain an x86-only primary distro. But that won't help address the longer term problems we will face. I'll spare the hyperbole for the moment, but I will add that this is a multi-year journey that we want to begin now. Yes, there are rough edges, yes this is cutting edge stuff, yes that is precisely what Fedora is all about.
Jon.
On Tue, Mar 20, 2012 at 12:54:36PM -0400, Jon Masters wrote:
The hardware is way slower ... so we can just build on faster hardware (x86_64). Which is the only sane way to do it. Trying to build on ARM directly is kind of a gimmick but nothing one can seriously use to build a whole operating system. (Yes it works but it is way to slow).
Well, we've done a number of mass rebuilds, a complete bootstrap from scratch, and several releases now. So, it might be a "gimmick", but it works. We need to stop thinking of ARM as it was 10 years ago. This year, we're going to see systems with 288+ cores in 2U of rack space.
Why are you even bringing up this as yet unreleased hardware in the context of "arm32 builds are slow" ? Even if it was released today, it doesn't solve this problem at all.
"Arm32 as primary" and "Arm64 as primary" are two entirely separate discussions, and conflating the two isn't solving anything.
Dave
On 03/20/2012 01:42 PM, Dave Jones wrote:
On Tue, Mar 20, 2012 at 12:54:36PM -0400, Jon Masters wrote:
The hardware is way slower ... so we can just build on faster hardware (x86_64). Which is the only sane way to do it. Trying to build on ARM directly is kind of a gimmick but nothing one can seriously use to build a whole operating system. (Yes it works but it is way to slow).
Well, we've done a number of mass rebuilds, a complete bootstrap from scratch, and several releases now. So, it might be a "gimmick", but it works. We need to stop thinking of ARM as it was 10 years ago. This year, we're going to see systems with 288+ cores in 2U of rack space.
Why are you even bringing up this as yet unreleased hardware in the context of "arm32 builds are slow" ? Even if it was released today, it doesn't solve this problem at all.
The hardware I'm citing there is 32-bit, and it's coming later *this year*. So I'm not conflating the two at all here, honestly :)
"Arm32 as primary" and "Arm64 as primary" are two entirely separate discussions, and conflating the two isn't solving anything.
I agree. Don't worry, we'll be getting to AArch64 later :)
Jon.
On Tue, 2012-03-20 at 17:37 +0100, drago01 wrote:
On Tue, Mar 20, 2012 at 5:34 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 09:21 AM, Ralf Corsepius wrote:
That said, I considera cross-building environment for secondary arch to be inevitable, which would at least help for the class of issues, I am referring to above.
I'm a big fan of cross compilation, but introducing it into Fedora in order to support ARM seems unlikely to succeed for too many reasons to go into.
The reasons are? ....
Fedora requires builds on the target system (either native or emulated, but not crossed) because: * some builds do tests, which will fail in cross * some builds make a mini-${FOO} and then execute that to build the final ${FOO}, which will fail in cross * BuildRequires must sometimes be of the same arch as the target (e.g., libraries) and sometimes of the same arch as the builder (tools) -- there is no way at present to know which is which, so it's a requirement that target arch == build arch
These can be overcome, but not without massive retooling.
-Chris
On Tue, Mar 20, 2012 at 4:58 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line? No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
Well the solution seems rather obvious to me . There is no real (technical) reason why you cannot build on x86_64 hardware. I never ever built anything on ARM directly using cross compilers on an x86_64 host is orders of magnitude faster so I saw no reason to attempt to build on ARM. The ARM hardware I worked with had only 128MB of RAM and a 400Mhz CPU but the same should apply to modern ARM platforms too (i.e building on x86_64 is just the saner way).
On Tue, Mar 20, 2012 at 08:58:45AM -0700, Brendan Conoboy wrote:
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line?
Is cross-compilation a realistic option? In theory this would improve the speed of builds to something like the x86-64 speed.
Rich.
Brendan Conoboy wrote:
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line?
IMHO, at MOST 50% longer (factor 1.5) build time, and that's already being nice. You're off by a factor of 4!
No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
That's exactly why we should stick with only x86 as primary architecture(s) in the foreseeable future.
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
See the other replies: chain builds, updates, platform-specific errors, build results. A lot of things depend on the builds to actually complete.
Kevin Kofler
On Tue, Mar 20, 2012 at 12:05 PM, Kevin Kofler kevin.kofler@chello.at wrote:
Brendan Conoboy wrote:
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line?
IMHO, at MOST 50% longer (factor 1.5) build time, and that's already being nice. You're off by a factor of 4!
No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
That's exactly why we should stick with only x86 as primary architecture(s) in the foreseeable future.
Only if you assume that high clock speed workloads are the only important workloads. For highly parallellizable tasks, an ARM system with tons of slower cores is a powerhouse. Think a db server serving huge numbers of queries.
-J
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
See the other replies: chain builds, updates, platform-specific errors, build results. A lot of things depend on the builds to actually complete.
Kevin Kofler
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Jon Ciesla wrote:
Only if you assume that high clock speed workloads are the only important workloads. For highly parallellizable tasks, an ARM system with tons of slower cores is a powerhouse. Think a db server serving huge numbers of queries.
Unfortunately, our builds are not that parallelizable, which is a major practical problem with ARM as a primary architecture.
As for supporting the use case for our users, I don't see why we cannot support those specialized tasks with a secondary architecture.
Kevin Kofler
On Tue, Mar 20, 2012 at 1:02 PM, Kevin Kofler kevin.kofler@chello.at wrote:
Jon Ciesla wrote:
Only if you assume that high clock speed workloads are the only important workloads. For highly parallellizable tasks, an ARM system with tons of slower cores is a powerhouse. Think a db server serving huge numbers of queries.
Unfortunately, our builds are not that parallelizable, which is a major practical problem with ARM as a primary architecture.
I was referring to use cases, not builds.
As for supporting the use case for our users, I don't see why we cannot support those specialized tasks with a secondary architecture.
Kevin Kofler
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Jon Ciesla wrote:
On Tue, Mar 20, 2012 at 1:02 PM, Kevin Kofler kevin.kofler@chello.at wrote:
Jon Ciesla wrote:
Only if you assume that high clock speed workloads are the only important workloads. For highly parallellizable tasks, an ARM system with tons of slower cores is a powerhouse. Think a db server serving huge numbers of queries.
Unfortunately, our builds are not that parallelizable, which is a major practical problem with ARM as a primary architecture.
I was referring to use cases, not builds.
As I wrote below:
As for supporting the use case for our users, I don't see why we cannot support those specialized tasks with a secondary architecture.
those use cases can be supported just fine by a secondary architecture.
Kevin Kofler
On 03/20/2012 11:58 AM, Brendan Conoboy wrote:
On 03/20/2012 08:24 AM, Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
Our current build systems can turn GCC 4.7 around in about 24 hours. The enterprise hardware we anticipate using will take that down to about 12 hours. If speed of build hardware is a consideration, where do you draw the line? No secondary arch is going to get to the speed of x86_64 in the foreseeable future, so it's effectively a way to keep PA an exclusive x86 club.
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
You're on the right track here, but you need to consider not just builds, but also updates. It's important for several reasons - an excessively long compile time means it's less likely that an update will be filed immediately after the build, for example. It's also important to consider things like incident response. For example, it's not as if we've never had a CVE filed against GCC that required rebuilding of packages. If it takes 24 hours to build gcc (a number I just made up for illustrative purposes), then you've got a scenario like where we're waiting on a build of gcc to get a libpng fix out so that firefox isn't being exploited. And sure, you're thinking "but who wants to run firefox on 32-bit arm?", but it doesn't matter, because it's being exploited for an extra day on x86 as well while we're waiting for libpng to be built with a newer compiler that isn't even in updates-testing yet.
That's the nightmare scenario, admittedly, but it still bears consideration.
On 3/20/12 8:58 AM, Brendan Conoboy wrote:
I think the real question is, for the developers of on devel-list, how will longer builds for one arch than another affect your workflow? If builds on two architectures start at the same time, but one takes longer to finish than the other, how will that impact you? Right now you'll still be able to see and use the results of the faster build before the slower build completes, so are you materially impacted?
You are materially impacted. AutoQA won't run until the entire build is complete. Updates cannot be prepared until the entire build is complete. Buildroots won't be updated with the build results until the entire build is complete. You won't know if your build /fails/ on the arch until it's done, etc...
Having one arch significantly slower than the others absolutely creates material impact upon developers.
On 03/20/2012 11:16 AM, Jesse Keating wrote:
You are materially impacted. AutoQA won't run until the entire build is complete. Updates cannot be prepared until the entire build is complete. Buildroots won't be updated with the build results until the entire build is complete. You won't know if your build /fails/ on the arch until it's done, etc...
Having one arch significantly slower than the others absolutely creates material impact upon developers.
I haven't run this by anybody yet, so if it's nonsense just say so, but...
Would it be reasonable to, even amongst primary architectures, allow these steps to go forward even if one arch fails while another succeeds? Let's say we have arch-groups in primary- i686 and x86_64 are in a group, armv7hl and armv5tel are in a group. The results of one group do not inhibit the progress of another. Feasible with a bit of retooling, or a nightmare waiting to happen? The discussion so far has focused almost exclusively on build time. We hear you. Let's talk about what to do about it. And what concerns there are besides build time.
On Tue, Mar 20, 2012 at 2:59 PM, Brendan Conoboy blc@redhat.com wrote:
On 03/20/2012 11:16 AM, Jesse Keating wrote:
You are materially impacted. AutoQA won't run until the entire build is complete. Updates cannot be prepared until the entire build is complete. Buildroots won't be updated with the build results until the entire build is complete. You won't know if your build /fails/ on the arch until it's done, etc...
Having one arch significantly slower than the others absolutely creates material impact upon developers.
I haven't run this by anybody yet, so if it's nonsense just say so, but...
Would it be reasonable to, even amongst primary architectures, allow these steps to go forward even if one arch fails while another succeeds? Let's say we have arch-groups in primary- i686 and x86_64 are in a group, armv7hl and armv5tel are in a group. The results of one group do not inhibit the progress of another. Feasible with a bit of retooling, or a nightmare waiting to happen? The discussion so far has focused almost exclusively on build time. We hear you. Let's talk about what to do about it. And what concerns there are besides build time.
You mean aside from what you already have as a secondary arch, which allows this explicitly?
What would you look to gain by having a half and half arch? All of the PR but none of the requirements for consistency and quality? I can sympathize with trying to be creative on solving a concern, but this probably is not how you want to go about it.
josh
On 3/20/12 11:59 AM, Brendan Conoboy wrote:
I haven't run this by anybody yet, so if it's nonsense just say so, but...
Would it be reasonable to, even amongst primary architectures, allow these steps to go forward even if one arch fails while another succeeds? Let's say we have arch-groups in primary- i686 and x86_64 are in a group, armv7hl and armv5tel are in a group. The results of one group do not inhibit the progress of another. Feasible with a bit of retooling, or a nightmare waiting to happen? The discussion so far has focused almost exclusively on build time. We hear you. Let's talk about what to do about it. And what concerns there are besides build time.
What you are describing is what we tried to do, which resulted in secondary arches. The primary arch group can move on with life, while the secondary arch plods along, hopefully finishing. If it doesn't finish, it can catch up later, but for the primary arches, life moves on.
So if you're willing to live like that, I must ask again, what do you think you'll be getting out of being a primary arch?
On 03/20/2012 12:05 PM, Jesse Keating wrote:
So if you're willing to live like that, I must ask again, what do you think you'll be getting out of being a primary arch?
I'm willing to temporarily do better than secondary and worse than primary on the road to becoming primary. This is a huge transition- identifying the right path to make that transition is part of what this is about. The whole point of this thread is to establish requirements for promotion. Part of that discussion logically includes the steps to get there. Currently what I hear is "be as good as x86 and you're there." That's not productive. There are legitimate issues with moving to PA so we're having this discussion to identify them and ultimately work through them.
On 3/20/12 12:14 PM, Brendan Conoboy wrote:
On 03/20/2012 12:05 PM, Jesse Keating wrote:
So if you're willing to live like that, I must ask again, what do you think you'll be getting out of being a primary arch?
I'm willing to temporarily do better than secondary and worse than primary on the road to becoming primary. This is a huge transition- identifying the right path to make that transition is part of what this is about. The whole point of this thread is to establish requirements for promotion. Part of that discussion logically includes the steps to get there. Currently what I hear is "be as good as x86 and you're there." That's not productive. There are legitimate issues with moving to PA so we're having this discussion to identify them and ultimately work through them.
What does "better than secondary arch" mean to you? I'm really struggling here.
We as a group have identified many of the roadblocks or pain points of bringing arm into primary arch. You're suggested solution in this sub-thread is effectively treating arm as secondary arch, and when asked about this, you've avoided the question, once again, of what it is you expect to get out of being primary arch.
I'm really not sure how much more rational discussion we can have here.
On 03/20/2012 12:19 PM, Jesse Keating wrote:
What does "better than secondary arch" mean to you? I'm really struggling here.
As an example, the same koji server handling x86 builds handling ARM builds. The same facilities providing power, cooling, storage. Subject to applicability, the same QE mechanisms being employed. The same release schedule. Comparable positioning on the Fedora downloads pages. Primary and secondary are night-and-day different, it isn't just the integrated build system being disconnected, it's end-to-end.
We as a group have identified many of the roadblocks or pain points of bringing arm into primary arch.
What pain points have been described other than "I am concerned about the impact of builds on the whole running slower than they do now"? This is not a facetious question, this is really what we're trying to get from the thread.
On 03/20/2012 03:32 PM, Brendan Conoboy wrote:
On 03/20/2012 12:19 PM, Jesse Keating wrote:
What does "better than secondary arch" mean to you? I'm really struggling here.
As an example, the same koji server handling x86 builds handling ARM builds. The same facilities providing power, cooling, storage. Subject to applicability, the same QE mechanisms being employed. The same release schedule. Comparable positioning on the Fedora downloads pages. Primary and secondary are night-and-day different, it isn't just the integrated build system being disconnected, it's end-to-end.
So, what we really need to talk about is how to make it possible to live as a secondary arch with stricter requirements than some secondaries have.
Ultimately here the question is how to make graduated transitions from one to the other. But the reasons to be a primary arch can't be as a mechanism to become good enough to be a primary arch.
Forgive me for an extended metaphor, but this is like minor league baseball. Nobody gets promoted to the majors to learn how to be good enough for the majors. Exactly the same situation exists between being a secondary arch vs a primary one - which means we need to do a much better job in secondary land of being able to prepare something for the time when it's appropriate to make it a primary arch. But promoting something to primary shouldn't be the mechanism to /prepare/ it for a higher level of expectations.
We as a group have identified many of the roadblocks or pain points of bringing arm into primary arch.
What pain points have been described other than "I am concerned about the impact of builds on the whole running slower than they do now"? This is not a facetious question, this is really what we're trying to get from the thread.
Also the things I've listed before involving shared maintainership of the whole as opposed to effectively having arch maintainers, the installation question, and the question of what the test plans look like. Right now those are questions you're taking back to work on, and I understand that, but I think it's probably the sort of thing people refer to when they say things like the above.
On 3/20/12 12:32 PM, Brendan Conoboy wrote:
On 03/20/2012 12:19 PM, Jesse Keating wrote:
What does "better than secondary arch" mean to you? I'm really struggling here.
As an example, the same koji server handling x86 builds handling ARM builds.
Only the koji hub would be the same, the arm builders would be different machines. This isn't all that different from having the primary hub trigger the arm hub to start a build on the arm builders.
The same facilities providing power, cooling, storage.
The PPC builders are there, or at least were at some point, why not propose moving some arm stuff there for the arm secondary arch effort?
Subject to applicability, the same QE mechanisms being employed.
I don't see SA/PA mattering as much here. It's up to QE what they want to take on and what they point automated tooling at.
The same release schedule.
That's set by you, as a secondary arch. Why not pin it to the Fedora release schedule and see how well you do?
Comparable positioning on the Fedora downloads pages.
That's a big ball of "another issue" there. That's a constant fight within the spins of the primary arch products, and from what I've seen, Arm products are more closely aligned with spins than anything else.
That said, within the websites group perhaps there is room for a featured secondary arch, with high visibility. Having something to point to first would help.
Primary and secondary are night-and-day different, it isn't just the integrated build system being disconnected, it's end-to-end.
We as a group have identified many of the roadblocks or pain points of bringing arm into primary arch.
What pain points have been described other than "I am concerned about the impact of builds on the whole running slower than they do now"? This is not a facetious question, this is really what we're trying to get from the thread.
Did you just ignore the emails starting these two threads by mjg and pjones? I believe they outlined some very good requirements for getting a secondary arch into primary. These longer threads have been debating some of the finer points of those proposals.
On 03/20/2012 01:03 PM, Jesse Keating wrote:
As an example, the same koji server handling x86 builds handling ARM builds.
Only the koji hub would be the same, the arm builders would be different machines. This isn't all that different from having the primary hub trigger the arm hub to start a build on the arm builders.
So in principle, do you object to the same koji hub being used for ARM if ARM is still SA?
The PPC builders are there, or at least were at some point, why not propose moving some arm stuff there for the arm secondary arch effort?
...
I don't see SA/PA mattering as much here. It's up to QE what they want to take on and what they point automated tooling at.
The sense I'm getting from your reply is that the PA/SA designation is almost decorative, that a secondary can do anything a primary can, except inhibit the progress of builds. So, if the Fedora ARM team fixes all broken builds, brings in all the QE mechanisms, engages the Fedora system at every level of the organization, solves the the build time performance issue, and otherwise keep at it until we're functionally indistinguishable from PA, *then* it's time to have the PA/SA discussion. Is that right? Speaking for myself, I see most of these things as a benefit of membership in PA rather than prerequisites. Or more to the point, transitioning from SA to PA means doing all of these things, so it's really just an order of operations that needs to be agreed upon.
That's set by you, as a secondary arch. Why not pin it to the Fedora release schedule and see how well you do?
We're quite close, though clearly the QE is different.
That said, within the websites group perhaps there is room for a featured secondary arch, with high visibility. Having something to point to first would help.
Fair enough.
Did you just ignore the emails starting these two threads by mjg and pjones? I believe they outlined some very good requirements for getting a secondary arch into primary. These longer threads have been debating some of the finer points of those proposals.
On the contrary, mjg and pjones' emails are just the sort of constructive and thoughtful feedback I'm looking for. If the points they've raised (which they also raised yesterday) speak to everybody's concerns, then I'm happy to view them as the authoritative feedback of Fedora-devel for our planning purposes. On the other hand, if they've left anything out that should be considered in this plan, I'd like to see it.
On Tue, Mar 20, 2012 at 02:33:57PM -0700, Brendan Conoboy wrote:
The sense I'm getting from your reply is that the PA/SA designation is almost decorative, that a secondary can do anything a primary can, except inhibit the progress of builds. So, if the Fedora ARM team fixes all broken builds, brings in all the QE mechanisms, engages the Fedora system at every level of the organization, solves the the build time performance issue, and otherwise keep at it until we're functionally indistinguishable from PA, *then* it's time to have the PA/SA discussion. Is that right? Speaking for myself, I see most of these things as a benefit of membership in PA rather than prerequisites. Or more to the point, transitioning from SA to PA means doing all of these things, so it's really just an order of operations that needs to be agreed upon.
I think you're looking at this in slightly the wrong way. Being a primary architecture isn't meant to be a benefit to the port - it's meant to be a benefit to Fedora. Adding arm to the PA list means you'll have to take on a huge number of additional responsibilities, deal with more people who are unhappy about the impact upon their packages and so on. You get very little out of it except that there's more people to tell you that something's broken. The question is whether making arm a primary architecture makes Fedora a better distribution, and yes, in an ideal world arm would demonstrate that it was just as functional as x86 before we had to make that decision.
The only reward you'll get from being a primary architecture is basking in the knowledge that the project thinks your work is good enough to be a primary architecture. The better you can demonstrate that in advance, the easier the process will be.
On Tue, Mar 20, 2012 at 10:58 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
I think you're looking at this in slightly the wrong way. Being a primary architecture isn't meant to be a benefit to the port - it's meant to be a benefit to Fedora. Adding arm to the PA list means you'll have to take on a huge number of additional responsibilities, deal with more people who are unhappy about the impact upon their packages and so on. You get very little out of it except that there's more people to tell you that something's broken.
I don't think this is true: On a primary architecture, every package maintainer is be expected to handle their own packages; this should actually significantly decrease the load on the "architecture maintainers". Mirek
On Wed, Mar 21, 2012 at 10:41:33AM +0100, Miloslav Trmač wrote:
On Tue, Mar 20, 2012 at 10:58 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
I think you're looking at this in slightly the wrong way. Being a primary architecture isn't meant to be a benefit to the port - it's meant to be a benefit to Fedora. Adding arm to the PA list means you'll have to take on a huge number of additional responsibilities, deal with more people who are unhappy about the impact upon their packages and so on. You get very little out of it except that there's more people to tell you that something's broken.
I don't think this is true: On a primary architecture, every package maintainer is be expected to handle their own packages; this should actually significantly decrease the load on the "architecture maintainers".
The expectation would be that the architecture maintainers have fixed everything before moving to being a primary architecture, so this should only be an issue if maintainers or upstream manage to come up with new breakage. But yes, it forces people to care about something they might previously have ignored, so I guess that's an advantage.
On Wed, Mar 21, 2012 at 7:39 AM, Matthew Garrett mjg59@srcf.ucam.org wrote:
On Wed, Mar 21, 2012 at 10:41:33AM +0100, Miloslav Trmač wrote:
On Tue, Mar 20, 2012 at 10:58 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
I think you're looking at this in slightly the wrong way. Being a primary architecture isn't meant to be a benefit to the port - it's meant to be a benefit to Fedora. Adding arm to the PA list means you'll have to take on a huge number of additional responsibilities, deal with more people who are unhappy about the impact upon their packages and so on. You get very little out of it except that there's more people to tell you that something's broken.
I don't think this is true: On a primary architecture, every package maintainer is be expected to handle their own packages; this should actually significantly decrease the load on the "architecture maintainers".
The expectation would be that the architecture maintainers have fixed everything before moving to being a primary architecture, so this should only be an issue if maintainers or upstream manage to come up with new breakage. But yes, it forces people to care about something they might previously have ignored, so I guess that's an advantage.
Except when people are forced to look at it, their solution was often ExcludeArch for PPC. As I said in the other thread, you cannot force people to care about an architecture they don't know or want to learn.
That's not to say there weren't a large number of people that _did_ try and fix things. It's just not a clear cut "this arch is primary so package maintainers will fix arch issues".
josh
On 03/21/2012 09:21 AM, Josh Boyer wrote:
Except when people are forced to look at it, their solution was often ExcludeArch for PPC. As I said in the other thread, you cannot force people to care about an architecture they don't know or want to learn.
That suggests we need a FTBFS-like nightly test that lets us know about new, unexpected ExcludeArches in the distro.
On Wed, Mar 21, 2012 at 1:52 PM, Peter Jones pjones@redhat.com wrote:
On 03/21/2012 09:21 AM, Josh Boyer wrote:
Except when people are forced to look at it, their solution was often ExcludeArch for PPC. As I said in the other thread, you cannot force people to care about an architecture they don't know or want to learn.
That suggests we need a FTBFS-like nightly test that lets us know about new, unexpected ExcludeArches in the distro.
TBH we need someone to take over the FTBFS things that Matt use to do, there's still 400+ packages that are FTBFS on x86 primary arch post the F-17 gcc 4.7 mass rebuild and even some going all the way back to the F-12 mass rebuilt (yes 3 mass rebuilds ago!). that number was actually much closer to 600 but the ARM team have fixed well over 100 of those on mainline because they were blocking what we were doing on ARM. Then once that is in place a means of tracking ExcludeArch/ExclusiveArch would be an excellent and very useful tool for all arches, primary or otherwise IMO.
Peter
On Wed, Mar 21, 2012 at 9:52 AM, Peter Jones pjones@redhat.com wrote:
On 03/21/2012 09:21 AM, Josh Boyer wrote:
Except when people are forced to look at it, their solution was often ExcludeArch for PPC. As I said in the other thread, you cannot force people to care about an architecture they don't know or want to learn.
That suggests we need a FTBFS-like nightly test that lets us know about new, unexpected ExcludeArches in the distro.
Possibly. There are ExcludeArch trackers that people are supposed to make their bugs block and that was normally sufficient to give the arch team a heads up. However, I'm sure there were packages that didn't have bugs filed like that.
josh
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wed, 21 Mar 2012 10:12:58 -0400 Josh Boyer jwboyer@gmail.com wrote:
On Wed, Mar 21, 2012 at 9:52 AM, Peter Jones pjones@redhat.com wrote:
On 03/21/2012 09:21 AM, Josh Boyer wrote:
Except when people are forced to look at it, their solution was often ExcludeArch for PPC. As I said in the other thread, you cannot force people to care about an architecture they don't know or want to learn.
That suggests we need a FTBFS-like nightly test that lets us know about new, unexpected ExcludeArches in the distro.
Possibly. There are ExcludeArch trackers that people are supposed to make their bugs block and that was normally sufficient to give the arch team a heads up. However, I'm sure there were packages that didn't have bugs filed like that.
the main issue is that we need to have tracking in place that doesnt require people do anything. because people dont do what they should. I see it all the time when dealing with mass rebuilds etc people do one or 2 of the steps to remove a package, but quite often do not do all 3. We do have plans to redo it to make it a single step. the more we can automate the tracking of it the better we will nknow the full extent of where things are. if we can cut down on what people have to do the more likely it will be that we have true representation of the data.
Dennis
On 3/21/12 6:52 AM, Peter Jones wrote:
On 03/21/2012 09:21 AM, Josh Boyer wrote:
Except when people are forced to look at it, their solution was often ExcludeArch for PPC. As I said in the other thread, you cannot force people to care about an architecture they don't know or want to learn.
That suggests we need a FTBFS-like nightly test that lets us know about new, unexpected ExcludeArches in the distro.
This sounds like the perfect use case for a SCM feature I haven't had the time to work on.
If all commit diffs are sent to a message bus by way of a git hook, then one consumer on the bus could be canning for additions of ExcludeArch. When these are discovered a bug could be filed automatically, or some group would get notified, basically something would happen, and we wouldn't depend on a human noticing the change or following policy to file a bug.
In the short term, if this is something we see high value in tracking, we can just add another git hook to do this directly, rather than relying on a message bus.
On Wed, Mar 21, 2012 at 11:39 AM, Matthew Garrett mjg59@srcf.ucam.org wrote:
On Wed, Mar 21, 2012 at 10:41:33AM +0100, Miloslav Trmač wrote:
On Tue, Mar 20, 2012 at 10:58 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
I think you're looking at this in slightly the wrong way. Being a primary architecture isn't meant to be a benefit to the port - it's meant to be a benefit to Fedora. Adding arm to the PA list means you'll have to take on a huge number of additional responsibilities, deal with more people who are unhappy about the impact upon their packages and so on. You get very little out of it except that there's more people to tell you that something's broken.
I don't think this is true: On a primary architecture, every package maintainer is be expected to handle their own packages; this should actually significantly decrease the load on the "architecture maintainers".
The expectation would be that the architecture maintainers have fixed everything before moving to being a primary architecture, so this should only be an issue if maintainers or upstream manage to come up with new breakage. But yes, it forces people to care about something they might previously have ignored, so I guess that's an advantage.
And we've already being doing that with the vast majority of issues already fixed and committed to mainline.
Peter
On Wed, Mar 21, 2012 at 01:26:58PM +0000, Peter Robinson wrote:
On Wed, Mar 21, 2012 at 11:39 AM, Matthew Garrett mjg59@srcf.ucam.org wrote:
The expectation would be that the architecture maintainers have fixed everything before moving to being a primary architecture, so this should only be an issue if maintainers or upstream manage to come up with new breakage. But yes, it forces people to care about something they might previously have ignored, so I guess that's an advantage.
And we've already being doing that with the vast majority of issues already fixed and committed to mainline.
Agreed. I just mean that it's not a terribly significant benefit to becoming a primary architecture.
On 3/20/12 2:33 PM, Brendan Conoboy wrote:
On 03/20/2012 01:03 PM, Jesse Keating wrote:
As an example, the same koji server handling x86 builds handling ARM builds.
Only the koji hub would be the same, the arm builders would be different machines. This isn't all that different from having the primary hub trigger the arm hub to start a build on the arm builders.
So in principle, do you object to the same koji hub being used for ARM if ARM is still SA?
I'm not really sure how to process that question. As a current secondary arch, the primary hub is still the trigger point for the vast majority of the builds that will happen for arm. A successful build on the primary hub will trigger (through koji-shadow) a build on the secondary arches. I'm sure you're (painfully) aware of this.
I don't understand how the same koji hub could be used for arm while arm is still SA differently than above.
The PPC builders are there, or at least were at some point, why not propose moving some arm stuff there for the arm secondary arch effort?
...
I don't see SA/PA mattering as much here. It's up to QE what they want to take on and what they point automated tooling at.
The sense I'm getting from your reply is that the PA/SA designation is almost decorative, that a secondary can do anything a primary can, except inhibit the progress of builds.
Mostly. It'll take effort on the part of the secondary arch to engage these other parts of the Fedora project to convince them to pay attention to your arch. If you were a primary, they're essentially forced to care. Enticing them to care as a secondary arch goes a long long way to show that you're ready to be a primary arch and that the promotion would not cause much ripple effect.
So, if the Fedora ARM team fixes all broken builds, brings in all the QE mechanisms, engages the Fedora system at every level of the organization, solves the the build time performance issue, and otherwise keep at it until we're functionally indistinguishable from PA, *then* it's time to have the PA/SA discussion. Is that right?
That does seem right. Just like the old adage, dress for the job you want, not the one you have, or do the work for the promotion you want, and you'll earn the promotion, the same applies here. Show that the secondary arch can function well as a primary arch without significant burden to the rest of the project and then it's a much easier decision to make the promotion.
Speaking for myself, I see most of these things as a benefit of membership in PA rather than prerequisites. Or more to the point, transitioning from SA to PA means doing all of these things, so it's really just an order of operations that needs to be agreed upon.
"doing all of these things" doesn't happen magically just because the board/fesco grants that ARM is suddenly a primary arch. If we made arm a primary arch tomorrow, you'd still have to solve all the above issues, only now you've got hundreds of very angry developers who's workflow is now severely interrupted, and they're all calling for your head. Doesn't it make more sense to solve these issues /before/ doing the promotion? Figure out how to make the car go 60mph before taking it onto the freeway, else you'll piss off all the other cars on the freeway.
That's set by you, as a secondary arch. Why not pin it to the Fedora release schedule and see how well you do?
We're quite close, though clearly the QE is different.
That said, within the websites group perhaps there is room for a featured secondary arch, with high visibility. Having something to point to first would help.
Fair enough.
Did you just ignore the emails starting these two threads by mjg and pjones? I believe they outlined some very good requirements for getting a secondary arch into primary. These longer threads have been debating some of the finer points of those proposals.
On the contrary, mjg and pjones' emails are just the sort of constructive and thoughtful feedback I'm looking for. If the points they've raised (which they also raised yesterday) speak to everybody's concerns, then I'm happy to view them as the authoritative feedback of Fedora-devel for our planning purposes. On the other hand, if they've left anything out that should be considered in this plan, I'd like to see it.
Fair enough, I honestly didn't think you had ignored it, and it was rude of me to suggest otherwise. I apologize for that.
"fedora-devel" doesn't really have anything of the "authoritative" sort, we just have a lot of subscribers and opinions. Those opinions are usually considered by FESCo when FESCo makes a decision, and generally considered by those proposing something and asking for feedback on their way to a FESCo decision.
On 03/20/2012 03:33 PM, Jesse Keating wrote:
So in principle, do you object to the same koji hub being used for ARM if ARM is still SA?
I'm not really sure how to process that question. As a current secondary arch, the primary hub is still the trigger point for the vast majority of the builds that will happen for arm. A successful build on the primary hub will trigger (through koji-shadow) a build on the secondary arches. I'm sure you're (painfully) aware of this. I don't understand how the same koji hub could be used for arm while arm is still SA differently than above.
Context for the question: One of the benefits to PA that I see is that PA's infrastructure is really good. I mean, the reliability is outstanding. Kudos to all involved. To what extent can a secondary sharing in that bliss? Machines in the same racks? Services on the same machines? The same koji hub, but treating failed SA builds differently than failed PA builds? How close can we get to treating a secondary architecture like a primary, as far as that infrastructure goes? Is the limit, again, as close as we want up to the point of it affecting primary programmatically? Or should there be a separation, even if it's technically feasible to combine the services for both SA and PA?
Mostly. It'll take effort on the part of the secondary arch to engage these other parts of the Fedora project to convince them to pay attention to your arch. If you were a primary, they're essentially forced to care. Enticing them to care as a secondary arch goes a long long way to show that you're ready to be a primary arch and that the promotion would not cause much ripple effect.
Agreed. We definitely want promotion, in the end, to be a small ripple, if little notice because the hard parts are already done.
That does seem right. Just like the old adage, dress for the job you want, not the one you have, or do the work for the promotion you want, and you'll earn the promotion, the same applies here. Show that the secondary arch can function well as a primary arch without significant burden to the rest of the project and then it's a much easier decision to make the promotion.
Yes...
"doing all of these things" doesn't happen magically just because the board/fesco grants that ARM is suddenly a primary arch. If we made arm a primary arch tomorrow, you'd still have to solve all the above issues, only now you've got hundreds of very angry developers who's workflow is now severely interrupted, and they're all calling for your head. Doesn't it make more sense to solve these issues /before/ doing the promotion? Figure out how to make the car go 60mph before taking it onto the freeway, else you'll piss off all the other cars on the freeway.
Absolutely. We're having this discussion as a way to solve the issues before promotion. None of us expect ARM to be promoted today, or tomorrow, but perhaps 3-5 months from now is realistic. Or maybe that's too soon. The bottom line is that there are issues to be worked out and that's what has prompted the discussion.
Fair enough, I honestly didn't think you had ignored it, and it was rude of me to suggest otherwise. I apologize for that.
Thanks- I didn't think your replies mean spirited and very much appreciate your taking the time to think about the issues with the proposal. It's a work in progress and it will certainly be a while before it's complete enough that to be approved. We're charting a course toward primary and these threads are simply a way of drawing up the map we'll be taking.
"fedora-devel" doesn't really have anything of the "authoritative" sort, we just have a lot of subscribers and opinions. Those opinions are usually considered by FESCo when FESCo makes a decision, and generally considered by those proposing something and asking for feedback on their way to a FESCo decision.
Yep. Regards,
On 03/20/2012 06:51 PM, Brendan Conoboy wrote:
On 03/20/2012 03:33 PM, Jesse Keating wrote:
"doing all of these things" doesn't happen magically just because the board/fesco grants that ARM is suddenly a primary arch. If we made arm a primary arch tomorrow, you'd still have to solve all the above issues, only now you've got hundreds of very angry developers who's workflow is now severely interrupted, and they're all calling for your head. Doesn't it make more sense to solve these issues /before/ doing the promotion? Figure out how to make the car go 60mph before taking it onto the freeway, else you'll piss off all the other cars on the freeway.
Absolutely. We're having this discussion as a way to solve the issues before promotion. None of us expect ARM to be promoted today, or tomorrow, but perhaps 3-5 months from now is realistic. Or maybe that's too soon. The bottom line is that there are issues to be worked out and that's what has prompted the discussion.
I think this is the best takeway from the thread I've seen so far. We're trying to figure out where to go to get to PA. If it turns out F18 is wildly optimistic, ok, no problem. We'll come back later after we get all the kinks ironed out. But the past few days have provided invaluable initial input as we figure out how to drive at 60mph, while also giving you greater than 60mpg in power/performance :)
Jon.
On Tue, 2012-03-20 at 13:03 -0700, Jesse Keating wrote:
Subject to applicability, the same QE mechanisms being employed.
I don't see SA/PA mattering as much here. It's up to QE what they want to take on and what they point automated tooling at.
In theory...yeah. In boring every day practice, we'd take a lot more heat for 'not QAing a primary arch' than we would for 'not QAing a secondary arch'.
I mean, right now, Fedora QA does just about zip for PPC or ARM. And no-one not directly involved in PPC or ARM has ever complained to us about that.
If ARM were a primary arch, I rather suspect they would.
But sure, in theory, we can do just about anything for a secondary arch that we do for a primary arch, I don't think there's any technical barrier to us doing update karma for ARM and test days for ARM and a release validation process for ARM and all the rest of it. It's just a question of motivation and personpower.
On 3/20/12 5:14 PM, Adam Williamson wrote:
But sure, in theory, we can do just about anything for a secondary arch that we do for a primary arch, I don't think there's any technical barrier to us doing update karma for ARM and test days for ARM and a release validation process for ARM and all the rest of it. It's just a question of motivation and personpower.
My point is that the motivation and personpower can come independent of whether arm is a PA or an SA. As you say, no technical barrier.
----- Original Message -----
From: "Brendan Conoboy" blc@redhat.com To: devel@lists.fedoraproject.org Sent: Tuesday, March 20, 2012 9:14:11 PM Subject: Re: RFC: Primary architecture promotion requirements
On 03/20/2012 12:05 PM, Jesse Keating wrote:
So if you're willing to live like that, I must ask again, what do you think you'll be getting out of being a primary arch?
I'm willing to temporarily do better than secondary and worse than primary on the road to becoming primary. This is a huge transition- identifying the right path to make that transition is part of what this is about. The whole point of this thread is to establish requirements for promotion. Part of that discussion logically includes the steps to get there. Currently what I hear is "be as good as x86 and you're there." That's not productive. There are legitimate issues with moving to PA so we're having this discussion to identify them and ultimately work through them.
If we really have to set requirements for proposals I see one thing totally missed from the discussion up to now. Is the SA ready? And giving a definition for being ready: * does it release together with the PAs? * has it ever released without a significant delay? define delay - 1 month?, 3 months? * does it have the majority of the packages readgy? 70? 80? 90%? * name yours
I really think that before promoting SA to PA it should have at least one release being done together with the PAs with a sufficient feature set. Nothing prevents SA to prove that it can deliver on time much like the PA do now.
Alex
-- Brendan Conoboy / Red Hat, Inc. / blc@redhat.com -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Jakub Jelinek wrote:
I think the speed of the build hardware should be also part of the criteria, as all primary architectures are built synchronously. GCC on x86_64/i686 currently builds often in 2 hours, sometimes in 4 hours if a slower or more busy box is chosen, but on ARM it regularly builds 2 days. That is a slow down factor of 12x-24x, guess for other larger packages it is similar.
+1
Kevin Kofler
On Tue, 2012-03-20 at 15:19 +0000, Matthew Garrett wrote:
This is very much a draft, but I'd like to start a discussion regarding what we expect from primary architectures. Feedback not only welcome, but actively desired.
In order to ensure that these expectations are met, secondary architectures must meet various criteria before they can be promoted:
...
- All supported platforms must have kernels built from the Fedora
kernel SRPM and enabled by default in the spec file. Each kernel must be built in a timely manner for every SRPM upload.
I do not like this requirement. This seems to be specifically provided to block the possibility to have ARM as a primary architecture if we do not want to support just one or two ARM platforms. I do not really see a problem in limiting platforms during rawhide development and branched development. Additional platforms could be enabled for final builds before the release freezes and for update builds.
Another solution might be in koji where the kernels for the additional platforms would be built in parallel on multiple build hosts. Of course that would require changes in koji.
Of course the general requirement that builds on the architecture to be promoted must not take much longer time than builds on the current primary architectures still stays.
On Tue, Mar 20, 2012 at 11:37 AM, Tomas Mraz tmraz@redhat.com wrote:
On Tue, 2012-03-20 at 15:19 +0000, Matthew Garrett wrote:
This is very much a draft, but I'd like to start a discussion regarding what we expect from primary architectures. Feedback not only welcome, but actively desired.
In order to ensure that these expectations are met, secondary architectures must meet various criteria before they can be promoted:
...
- All supported platforms must have kernels built from the Fedora
kernel SRPM and enabled by default in the spec file. Each kernel must be built in a timely manner for every SRPM upload.
I do not like this requirement. This seems to be specifically provided to block the possibility to have ARM as a primary architecture if we do not want to support just one or two ARM platforms. I do not really see a problem in limiting platforms during rawhide development and branched development. Additional platforms could be enabled for final builds before the release freezes and for update builds.
There's nothing blocking ARM from building multiple kernels in that requirement. They just need to all be enabled in the SRPM that gets sent to koji for the build. We do this for 32-bit x86 already by building both the normal and PAE i686 variants. The intention is to basically have a consistent and reproducible build for all kernels built from a single SRPM.
I really don't think we want to enable additional kernels for final builds that have not been tested at all during development of the release. That doesn't make sense at all and sounds like poor engineering practice.
Another solution might be in koji where the kernels for the additional platforms would be built in parallel on multiple build hosts. Of course that would require changes in koji.
That would be acceptable of course, and it would still fit with the requirement above just fine.
josh
On 03/20/2012 08:47 AM, Josh Boyer wrote:
There's nothing blocking ARM from building multiple kernels in that requirement. They just need to all be enabled in the SRPM that gets sent to koji for the build. We do this for 32-bit x86 already by building both the normal and PAE i686 variants. The intention is to basically have a consistent and reproducible build for all kernels built from a single SRPM.
This is infact what we're doing now- a single kernel build produces rpms for a number of different ARM platforms (omap, tegra, imx, versatile, etc).
I really don't think we want to enable additional kernels for final builds that have not been tested at all during development of the release. That doesn't make sense at all and sounds like poor engineering practice.
Agree.
Another solution might be in koji where the kernels for the additional platforms would be built in parallel on multiple build hosts. Of course that would require changes in koji.
That would be acceptable of course, and it would still fit with the requirement above just fine.
I'm not sure how it would work, but if koji can be extended to distribute a single arch build across multiple systems where an identical srpm can be built with a koji-controlled set of flags, this would take care of the wide-breadth of kernels needing to be built.
We've also had some success with distcc, but have not proposed using it as reproducability of builds becomes an issue.
On 3/20/12 9:38 AM, Brendan Conoboy wrote:
I'm not sure how it would work, but if koji can be extended to distribute a single arch build across multiple systems where an identical srpm can be built with a koji-controlled set of flags, this would take care of the wide-breadth of kernels needing to be built.
We've also had some success with distcc, but have not proposed using it as reproducability of builds becomes an issue.
We had something like this where i586 and i686 were considered different arches, at least for the kernel, and those two builds would happen in parallel often on different machines. Perhaps the same could be done for the arm variants as well.
On 03/20/2012 11:37 AM, Jesse Keating wrote:
We had something like this where i586 and i686 were considered different arches, at least for the kernel, and those two builds would happen in parallel often on different machines. Perhaps the same could be done for the arm variants as well.
Right now we consider armv5tel and armv7hl to be different aches so they are built on separate machines. That bit of optimization is done.
IIf there is some sane way to distribute a single armv7hl or armv5tel build across multiple builders that may be an interesting avenue to pursue (Sanity tbd by releng:-). The builders we expect to see this year have 4 cores, but if we could realistically do a 'make -j 32' and have 8 systems involved in any one package, that'd certainly take care of performance concerns for parallelizable builds. It's a neat idea, but making it work reliably is a proposal unto itself.
On 3/20/12 12:05 PM, Brendan Conoboy wrote:
IIf there is some sane way to distribute a single armv7hl or armv5tel build across multiple builders that may be an interesting avenue to pursue (Sanity tbd by releng:-). The builders we expect to see this year have 4 cores, but if we could realistically do a 'make -j 32' and have 8 systems involved in any one package, that'd certainly take care of performance concerns for parallelizable builds. It's a neat idea, but making it work reliably is a proposal unto itself.
Yeah, I see that as another rather large proposal. It could make some other build tasks go faster too, even on x86, but traditionally we've pushed away from that because of the complexity it presents and concerns of reproducible results. Then again this discussion was ages ago when the tech to do such things was youngish.
On Tue, Mar 20, 2012 at 04:37:17PM +0100, Tomas Mraz wrote:
On Tue, 2012-03-20 at 15:19 +0000, Matthew Garrett wrote:
- All supported platforms must have kernels built from the Fedora
kernel SRPM and enabled by default in the spec file. Each kernel must be built in a timely manner for every SRPM upload.
I do not like this requirement. This seems to be specifically provided to block the possibility to have ARM as a primary architecture if we do not want to support just one or two ARM platforms. I do not really see a problem in limiting platforms during rawhide development and branched development. Additional platforms could be enabled for final builds before the release freezes and for update builds.
The problem with not doing a full set of builds in rawhide is that it significantly reduces the testing - both in terms of making sure that code still bilds, and in terms of making sure that we don't have unexpected functional regressions. Shortly before release is a really bad time to discover this. My impression is that the kernel team don't want that to be a possibility.
Another solution might be in koji where the kernels for the additional platforms would be built in parallel on multiple build hosts. Of course that would require changes in koji.
Yes, there's no fundamental reason that these builds need to occur in series. If koji had support for exploding builds out to multiple machines then things would work much better. Another alternative might be to investigate whether distcc is practical in this environment.
On Tue, Mar 20, 2012 at 10:51 AM, Matthew Garrett mjg59@srcf.ucam.org wrote:
On Tue, Mar 20, 2012 at 04:37:17PM +0100, Tomas Mraz wrote:
On Tue, 2012-03-20 at 15:19 +0000, Matthew Garrett wrote:
- All supported platforms must have kernels built from the Fedora
kernel SRPM and enabled by default in the spec file. Each kernel must be built in a timely manner for every SRPM upload.
I do not like this requirement. This seems to be specifically provided to block the possibility to have ARM as a primary architecture if we do not want to support just one or two ARM platforms. I do not really see a problem in limiting platforms during rawhide development and branched development. Additional platforms could be enabled for final builds before the release freezes and for update builds.
The problem with not doing a full set of builds in rawhide is that it significantly reduces the testing - both in terms of making sure that code still bilds, and in terms of making sure that we don't have unexpected functional regressions. Shortly before release is a really bad time to discover this. My impression is that the kernel team don't want that to be a possibility.
Another solution might be in koji where the kernels for the additional platforms would be built in parallel on multiple build hosts. Of course that would require changes in koji.
Yes, there's no fundamental reason that these builds need to occur in series. If koji had support for exploding builds out to multiple machines then things would work much better. Another alternative might be to investigate whether distcc is practical in this environment.
Matthew, can you add your initial list to the ticket as well, so we have these starting places to refer to?
-J
-- Matthew Garrett | mjg59@srcf.ucam.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Tue, Mar 20, 2012 at 10:55:41AM -0500, Jon Ciesla wrote:
Matthew, can you add your initial list to the ticket as well, so we have these starting places to refer to?
I was planning to after we'd had some discussion here, just to make sure I wasn't proposing anything too unreasonable.
On Tue, Mar 20, 2012 at 10:57 AM, Matthew Garrett mjg59@srcf.ucam.org wrote:
On Tue, Mar 20, 2012 at 10:55:41AM -0500, Jon Ciesla wrote:
Matthew, can you add your initial list to the ticket as well, so we have these starting places to refer to?
I was planning to after we'd had some discussion here, just to make sure I wasn't proposing anything too unreasonable.
Excellent.
-- Matthew Garrett | mjg59@srcf.ucam.org -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Tomas Mraz wrote:
On Tue, 2012-03-20 at 15:19 +0000, Matthew Garrett wrote:
- All supported platforms must have kernels built from the Fedora
kernel SRPM and enabled by default in the spec file. Each kernel must be built in a timely manner for every SRPM upload.
I do not like this requirement. This seems to be specifically provided to block the possibility to have ARM as a primary architecture if we do not want to support just one or two ARM platforms. I do not really see a problem in limiting platforms during rawhide development and branched development. Additional platforms could be enabled for final builds before the release freezes and for update builds.
Yet that requirement makes a lot of sense, and is yet another reason why ARM shouldn't even be CONSIDERED for primary status at this point. Building a separate kernel for every single machine just doesn't scale. Imagine if we had to build a Thinkpad kernel, a MacBook kernel, a Dell Inspiron kernel etc. (and I didn't even bring model numbers in here!). There's no way such a setup is supportable.
Another solution might be in koji where the kernels for the additional platforms would be built in parallel on multiple build hosts. Of course that would require changes in koji.
Indeed it would, and it still wouldn't fix the underlying issue.
Of course the general requirement that builds on the architecture to be promoted must not take much longer time than builds on the current primary architectures still stays.
Right, and I don't see ARM satisfying this any time soon either.
Kevin Kofler
On 3/20/12 8:37 AM, Tomas Mraz wrote:
I do not like this requirement. This seems to be specifically provided to block the possibility to have ARM as a primary architecture if we do not want to support just one or two ARM platforms. I do not really see a problem in limiting platforms during rawhide development and branched development. Additional platforms could be enabled for final builds before the release freezes and for update builds.
That just means those platforms are getting tested at the same time the rest of the distribution is, and then when you turn it on you suddenly find bugs that need to be fixed which invalidates testing done already. Playing the "turn it on late" game is a non-starter IMHO.
Matthew Garrett wrote:
This is very much a draft, but I'd like to start a discussion regarding what we expect from primary architectures. Feedback not only welcome, but actively desired.
So, first of all, I disagree that there should be a process for promoting an architecture to primary in the first place. The set of primary architectures (x86_64, i686) should be set in stone unless a MAJOR change in hardware landscape happens, e.g. x86 gets discontinued by the hardware manufacturers and everyone uses ARM instead. I don't see that happening any time soon. Adding primary architectures puts a major burden on all Fedora maintainers.
In the current state of things, I don't see a sufficient demand for making ARM (or even less any other secondary architecture) a primary architecture. If ARM is really the future as its proponents claim, we can revisit that in a few years. Not now.
The focus should be on finding ways to make secondary architecture releases more timely (i.e. it's not acceptable that e.g. the stable ARM release is still Fedora 14 which doesn't even get security updates anymore), not to cheat around the problem by making ARM a primary architecture (which does not help all the other secondary architectures).
Secondary architectures in Fedora are subject to looser constraints than primary architectures for two primary reasons:
- To make it easier to bootstrap an architecture without the overhead
of the primary architecture release engineering process 2) To avoid primary architecture development being held up by poorly developed or niche architectures
3) To avoid slowing down all builds by having to wait for the slow builders to complete them. 4) To avoid build failures caused by platform-specific toolchain bugs or limitations failing the entire build and forcing the maintainer to fix them.
In order to ensure that these expectations are met, secondary architectures must meet various criteria before they can be promoted:
- There must be adequate representation for the architecture on the
Fedora infrastructure and release engineering teams. 2) All builds must occur on Fedora-maintained build servers. 3) Where technically possible, all supported hardware targets must be supported via Anaconda. Exceptions are limited to highly resource constrained devices or hardware which provides no means for simultaneous support of install and target media.
Why would we want to make such an architecture (the "exceptions") primary?!
- All supported platforms must have kernels built from the Fedora
kernel SRPM and enabled by default in the spec file. Each kernel must be built in a timely manner for every SRPM upload. 5) Sufficient developer resources must be available to fix any architecture-specific issues in such a way that they do not delay overall Fedora development. 6) It must be possible for maintainers of critical-path hardware dependent packages to have direct access to supported hardware in order to rectify any release-blocking issues. For example, X maintainers must have direct access to any hardware with graphics capabilities. 7) The port must not rely on sourceless binaries unless they fall under the generic firmware exemption. Where source and toolchain are available, the binaries must be built in the Fedora build infrastructure.
This lacks several important requirements: * The platform should actually have a significant market share. Why would we want to make an obscure niche device a primary architecture (even if it fulfills all the 7 requirements you listed)? Your criteria would even allow architectures to become primary which don't have anywhere near the market share even ARM has (let alone x86). * The builders should not take significantly longer to complete builds than the x86 builders. Otherwise builds (especially chain builds) will be slowed down by a lot. * The architecture needs to have sufficient support from the pool of Fedora maintainers as a whole, who will be on the hook for making their packages build for it. * The toolchain (port) needs to have sufficient quality to not cause tons of platform-specific bugs which are of no fault of the package. (We've had our fair share of those with ppc/ppc64, due to both bugs and bizarre TOC size limitations.)
Kevin Kofler
I'm planning on moving this to the Wiki (as a draft) at the end of the week, so if people have any further feedback please let me know.
On Wed, Mar 28, 2012 at 10:11:35PM +0100, Matthew Garrett wrote:
I'm planning on moving this to the Wiki (as a draft) at the end of the week, so if people have any further feedback please let me know.
Now at http://fedoraproject.org/wiki/Secondary_Architecture_Promotion_Requirements_...
On Mon, Apr 2, 2012 at 4:45 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
Now at http://fedoraproject.org/wiki/Secondary_Architecture_Promotion_Requirements_...
FESCo would welcome more discussion of this draft, and plans to vote on it next week. Mirek
After some tweaking, these are now accepted as https://fedoraproject.org/w/index.php?title=Secondary_Architecture_Promotion...
On 04/23/2012 07:00 PM, Matthew Garrett wrote:
After some tweaking, these are now accepted as https://fedoraproject.org/w/index.php?title=Secondary_Architecture_Promotion...
Fail to see the reasoning why Anaconda and the "Installer team" are involved in these requirements care to elaborate how/why FESCo fits them into the bigger picture?
JBG
On Mon, Apr 23, 2012 at 07:33:44PM +0000, "Jóhann B. Guðmundsson" wrote:
On 04/23/2012 07:00 PM, Matthew Garrett wrote:
After some tweaking, these are now accepted as https://fedoraproject.org/w/index.php?title=Secondary_Architecture_Promotion...
Fail to see the reasoning why Anaconda and the "Installer team" are involved in these requirements care to elaborate how/why FESCo fits them into the bigger picture?
Because if you have hardware that can install via Anaconda and you don't support installing via Anaconda, you're not Fedora.
On 04/23/2012 07:42 PM, Matthew Garrett wrote:
Because if you have hardware that can install via Anaconda and you don't support installing via Anaconda, you're not Fedora.
So FESCo is in otherwords saying that other installers and even installing methods ( think like the distribution would be flashed to a device in the maybe not to distant future instead of being installed in the traditional sense as we know it to be ) are not allowed or otherwise considered to be part of the distribution should some community members want to writer and or package an alternative installer to ship and use on their spin?
JBG
2012/4/23 "Jóhann B. Guðmundsson" johannbg@gmail.com:
On 04/23/2012 07:42 PM, Matthew Garrett wrote:
Because if you have hardware that can install via Anaconda and you don't support installing via Anaconda, you're not Fedora.
So FESCo is in otherwords saying that other installers and even installing methods ( think like the distribution would be flashed to a device in the maybe not to distant future instead of being installed in the traditional sense as we know it to be ) are not allowed or otherwise considered to be part of the distribution should some community members want to writer and or package an alternative installer to ship and use on their spin?
I think it wasn't so much forbidding alternate methods as requiring that the primary one be supported.
-J
JBG
devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On 04/23/2012 12:54 PM, "Jóhann B. Guðmundsson" wrote:
So FESCo is in otherwords saying that other installers and even installing methods ( think like the distribution would be flashed to a device in the maybe not to distant future instead of being installed in the traditional sense as we know it to be ) are not allowed or otherwise considered to be part of the distribution should some community members want to writer and or package an alternative installer to ship and use on their spin?
It's not about anaconda specifically, it's about having a standard installer experience across all PAs to the extent technically sensible. Maybe something else will supplant anaconda in time.
On Mon, 2012-04-23 at 12:59 -0700, Brendan Conoboy wrote:
It's not about anaconda specifically, it's about having a standard installer experience across all PAs to the extent technically sensible. Maybe something else will supplant anaconda in time.
FWIW, in writing the QA release criteria, we used the generic term 'the installer' rather than the specific 'anaconda' to avoid this kind of ambiguity. In general I tend to prefer the use of generic terms in this kind of policy document for exactly this reason - to acknowledge and protect against the possibility of the currently-favoured implementation of any particular thing changing in future.
On Mon, Apr 23, 2012 at 07:54:57PM +0000, "Jóhann B. Guðmundsson" wrote:
On 04/23/2012 07:42 PM, Matthew Garrett wrote:
Because if you have hardware that can install via Anaconda and you don't support installing via Anaconda, you're not Fedora.
So FESCo is in otherwords saying that other installers and even installing methods ( think like the distribution would be flashed to a device in the maybe not to distant future instead of being installed in the traditional sense as we know it to be ) are not allowed or otherwise considered to be part of the distribution should some community members want to writer and or package an alternative installer to ship and use on their spin?
Fesco is saying that if you have hardware that can install via Anaconda, you must support installing via Anaconda. It's legitimate for you to also have other install mechanisms, and hardware that's incapable of supporting Anaconda installs isn't required to have them.
On Mon, Apr 23, 2012 at 4:00 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
Fesco is saying that if you have hardware that can install via Anaconda, you must support installing via Anaconda. It's legitimate for you to also have other install mechanisms, and hardware that's incapable of supporting Anaconda installs isn't required to have them.
Thanks for the clarification. I just wanted to make sure I understood that.
-- Jared Smith
On 04/23/2012 08:14 PM, Jared K. Smith wrote:
Fesco is saying that if you have hardware that can install via Anaconda, you must support installing via Anaconda. It's legitimate for you to also have other install mechanisms, and hardware that's incapable of supporting Anaconda installs isn't required to have them.
Thanks for the clarification. I just wanted to make sure I understood that.
FESCo should make that more clear in the requirements but even if they do they still make secondary architecture solely depended upon the will and the time of someone within the "Installer team" to implement the solution required to install Fedora for their architecture before they can become primary architecture.
That can mean from never to having to wait for several release cycles before becoming primary architecture for the distribution.
From my point of view that makes absolutly no sense and the requirements should be refactored to require an working installation method...
JBG
On Mon, Apr 23, 2012 at 08:57:47PM +0000, "Jóhann B. Guðmundsson" wrote:
On 04/23/2012 08:14 PM, Jared K. Smith wrote:
Fesco is saying that if you have hardware that can install via Anaconda, you must support installing via Anaconda. It's legitimate for you to also have other install mechanisms, and hardware that's incapable of supporting Anaconda installs isn't required to have them.
Thanks for the clarification. I just wanted to make sure I understood that.
FESCo should make that more clear in the requirements but even if they do they still make secondary architecture solely depended upon the will and the time of someone within the "Installer team" to implement the solution required to install Fedora for their architecture before they can become primary architecture.
Yes, in the same way that they need someone in the kernel team to build them a kernel. It's a package. It's under active development. It just needs someone on the architecture to write the code.
"Jóhann B. Guðmundsson" (johannbg@gmail.com) said:
On 04/23/2012 08:14 PM, Jared K. Smith wrote:
Fesco is saying that if you have hardware that can install via Anaconda, you must support installing via Anaconda. It's legitimate for you to also have other install mechanisms, and hardware that's incapable of supporting Anaconda installs isn't required to have them.
Thanks for the clarification. I just wanted to make sure I understood that.
FESCo should make that more clear in the requirements but even if they do they still make secondary architecture solely depended upon the will and the time of someone within the "Installer team" to implement the solution required to install Fedora for their architecture before they can become primary architecture.
There are these magical things called patches that can be submitted. Much like the secondary arch team would need to do so for the kernel (also mentioned in the guidelines), the X maintainers (also mentioned in the guidelines), etc. Unless you're suggesting secondary arch maintainers are somehow unable to do so?
Fedora's about providing a consistent experience wherever possible; this means using consistent interactive installation tools, consistent image creation tools, and so on.
Bill
On 04/23/2012 09:07 PM, Bill Nottingham wrote:
Fedora's about providing a consistent experience wherever possible; this means using consistent interactive installation tools
It's not enough to always be using the same tools but those tools need to be consitent in usage as well so for your noble goal now try putting consistency and Anaconda in the same sentence =)
All jokes aside and focusing on a bit more broader but fundemental question so I and perhaps some others can get a better understanding why things are as they are in the 21 first century.
Why do we distinquish between architectures in the first place and what do we hope to achive or otherwise gain from doing that in a community that first and foremost is about scratching your own itch ( at least for some of us )?
What is the justification for the need for the seperation in the firstplace?
JBG
On Mon, Apr 23, 2012 at 10:08:54PM +0000, "Jóhann B. Guðmundsson" wrote:
What is the justification for the need for the seperation in the firstplace?
You'd be fine with Fedora m68k? We have the separation because it's not just about scratching your own itch. Each additional supported architecture means that teams who are already overworked have to look after even more moving parts. Architecture-specific bugs are a pain for package maintainers to deal with. Attaching the Fedora name to poorly maintained ports weakens our brand. If we're going to support an architecture then its maintainers need to prove that they can maintain it.
On Mon, Apr 23, 2012 at 3:42 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
Because if you have hardware that can install via Anaconda and you don't support installing via Anaconda, you're not Fedora.
Just for the sake of argument, our Amazon EC2 images aren't using Anaconda for installation, but they're still considered part of Fedora.
I agree with the sentiment that Anaconda should be a requirement for instances where it makes sense to boot from bootable media (DVD/CD/USB) and install via Anaconda, but in the case of ARM, the traditional installation method is often "copy an image to an SD (or micro-SD) card and then boot from that card. Just to make sure I'm clear, are you insisting that the software tool that puts the image on the SD card be Anaconda, or are you OK with some other Fedora-approved tool for doing so?
-- Jared Smith
On Mon, Apr 23, 2012 at 3:13 PM, Jared K. Smith jsmith@fedoraproject.org wrote:
On Mon, Apr 23, 2012 at 3:42 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
Because if you have hardware that can install via Anaconda and you don't support installing via Anaconda, you're not Fedora.
Just for the sake of argument, our Amazon EC2 images aren't using Anaconda for installation, but they're still considered part of Fedora.
I think that's more akin to a spin.
I agree with the sentiment that Anaconda should be a requirement for instances where it makes sense to boot from bootable media (DVD/CD/USB) and install via Anaconda, but in the case of ARM, the traditional installation method is often "copy an image to an SD (or micro-SD) card and then boot from that card. Just to make sure I'm clear, are you insisting that the software tool that puts the image on the SD card be Anaconda, or are you OK with some other Fedora-approved tool for doing so?
-- Jared Smith -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Mon, Apr 23, 2012 at 04:13:40PM -0400, Jared K. Smith wrote:
On Mon, Apr 23, 2012 at 3:42 PM, Matthew Garrett mjg59@srcf.ucam.org wrote:
Because if you have hardware that can install via Anaconda and you don't support installing via Anaconda, you're not Fedora.
Just for the sake of argument, our Amazon EC2 images aren't using Anaconda for installation, but they're still considered part of Fedora.
That's why I recently added EC2 support to livemedia-creator. Since I don't have an EC2 account it is untested -- help would be appreciated.
I agree with the sentiment that Anaconda should be a requirement for instances where it makes sense to boot from bootable media (DVD/CD/USB) and install via Anaconda, but in the case of ARM, the traditional installation method is often "copy an image to an SD (or micro-SD) card and then boot from that card. Just to make sure I'm clear, are you insisting that the software tool that puts the image on the SD card be Anaconda, or are you OK with some other Fedora-approved tool for doing so?
We should be able to make images using livemedia-creator -- It needs to be run on the target hardware though, currently there is no cross-arch support. The last I heard from ARM was that Anaconda wasn't built for ARM.
The goal of creating lmc was to use Anaconda's logic for all installs, including creating system images or live media.. This will increase reliability and cut down on the number of bugs we see because livecd-creator really is just a wrapper around a yum chroot install.
On Tue, Apr 24, 2012 at 1:05 PM, Brian C. Lane bcl@redhat.com wrote:
That's why I recently added EC2 support to livemedia-creator. Since I don't have an EC2 account it is untested -- help would be appreciated.
Awesome. I'll try to check it out in the next week or so.
We should be able to make images using livemedia-creator -- It needs to be run on the target hardware though, currently there is no cross-arch support. The last I heard from ARM was that Anaconda wasn't built for ARM.
I know the folks up at Seneca have been working on building it for ARM -- I've been a bit out of the ARM loop the last couple of weeks, so I don't know the very latest status. I'll try to find out in tomorrow's ARM meeting.
The goal of creating lmc was to use Anaconda's logic for all installs, including creating system images or live media.. This will increase reliability and cut down on the number of bugs we see because livecd-creator really is just a wrapper around a yum chroot install.
Yes, I'm well aware of the goals behind lmc, and I think it's definitely a better way forward.
-- Jared Smith
"Jóhann B. Guðmundsson" (johannbg@gmail.com) said:
On 04/23/2012 07:00 PM, Matthew Garrett wrote:
After some tweaking, these are now accepted as https://fedoraproject.org/w/index.php?title=Secondary_Architecture_Promotion...
Fail to see the reasoning why Anaconda and the "Installer team" are involved in these requirements care to elaborate how/why FESCo fits them into the bigger picture?
We shouldn't be promoting anything to primary arch that you can't install.
Bill
On 04/23/2012 07:45 PM, Bill Nottingham wrote:
We shouldn't be promoting anything to primary arch that you can't install.
Valid point but it still does not explain why FESCo chose to limit that exclusively to Anaconda and the "Installer team" and their installation methods or lack there of.
<FESCo> Sorry guys your arch will have to wait to become primary until Anaconda and the "Installer team" has
a) Decided b) Designed c) Implemented
How to install your arch.
<Wanting to be primary arch SIG>
But our user base already can install via this method here and we actually preferred they did..
<FESCo> Sorry no flag no country those are the rules so if you want to sail under the Fedora flag you have to ride the snake...
JBG
On Mon, Apr 23, 2012 at 08:29:59PM +0000, "Jóhann B. Guðmundsson" wrote:
On 04/23/2012 07:45 PM, Bill Nottingham wrote:
We shouldn't be promoting anything to primary arch that you can't install.
Valid point but it still does not explain why FESCo chose to limit that exclusively to Anaconda and the "Installer team" and their installation methods or lack there of.
ARM is moving into the server and laptop space. There's an expectation that devices of that nature can be installed using Anaconda. If a port is only supporting embedded devices where Anaconda makes no sense then that's fine, but if it's supporting other hardware classes then it needs to have a working Anaconda. I've no idea why this is controversial.
devel@lists.stg.fedoraproject.org