On Wed, 28 Aug 2013 11:21:06 -0400 Jay Greguske jgregusk@redhat.com wrote:

Are the buildvm builders the bare-metal image builders?

No, as the name would suggest they are virtual instances. ;)

The buildvmhosts are still rhel6, but the virtual instances/buildvm's are now Fedora 19 (to match up with the Fedora 19 arm builders).

kevin

On Wed, 28 Aug 2013 18:22:57 +0300 Panu Matilainen pmatilai@laiskiainen.org wrote:

Hum... a question, or perhaps more like two:

Are you planning to move the remaining RHEL-6 builders to Fedora too, and if so, is this (builders running on Fedora ~latest) going to be a permament arrangement?

Quite possibly. Left is a pair of buildhw (hardware builders), buildppc (power7 boxes, mostly for epel6), and bkernel (kernel and secureboot builders). We are going to look and see if those can be moved over too without pain.

yeah, I think the plan is to do this moving forward (Fedora latest that is).

If the answer to both is "yes" then this is wonderful news for the package-management department.

It's good for infrastructure as well, in that we don't need to carry any 'special' rpm or anything if we need to change formats or the like.

Of course it does mean we could run into breakage if there's breakage in Fedora land, but thats good to know and fix too.

kevin

On Wed, 28 Aug 2013 11:41:31 -0400 Colin Walters walters@verbum.org wrote:

I think the most practical thing is to build version N of self-hosting systems using version N-1 (in addition to supporting building N with N). If you are using the latest to build the latest, you can more easily get into unstable states.

For example, if a new version of the Fedora kernel broke virtio-net, and you immediately upgrade the build VMs, then the system falls apart and requires manual system administrator intervention - you can't build a new fixed kernel.

Sure.

For example, the builders should stay Fedora 19 until Fedora 20 is officially released, then we will switch them to 20, etc.

Presumably version N-1 is more stable, although this is somewhat obviated by the Fedora kernel maintenance model of keeping all branches in sync.

We intend to re-install builders as needed, not during freezes and probably in a rolling manner. So any breakage should be isolated to non freeze windows.

kevin

On 08/28/2013 06:35 PM, Kevin Fenzi wrote:

On Wed, 28 Aug 2013 18:22:57 +0300 Panu Matilainen pmatilai@laiskiainen.org wrote:

...

Hum... a question, or perhaps more like two:

Are you planning to move the remaining RHEL-6 builders to Fedora too, and if so, is this (builders running on Fedora ~latest) going to be a permament arrangement?

Quite possibly. Left is a pair of buildhw (hardware builders), buildppc (power7 boxes, mostly for epel6), and bkernel (kernel and secureboot builders). We are going to look and see if those can be moved over too without pain.

yeah, I think the plan is to do this moving forward (Fedora latest that is).

Awesome. Just awesome.

...

If the answer to both is "yes" then this is wonderful news for the package-management department.

It's good for infrastructure as well, in that we don't need to carry any 'special' rpm or anything if we need to change formats or the like.

From package-management POV this makes it *possible* to meaningfully introduce certain types of new features. The builders being stuck on several years old, practically frozen version of rpm (and yum etc) has been one of, or perhaps *the* biggest roadblock and demotivating aspect in rpm development:

1) implement installation-related feature X in upstream git 2) wait for it to make its way next stable upstream release 3) wait for the upstream release to make its way into Fedora 4) grow many more gray hairs while waiting for next major RHEL version 5) wait a bit more for the builders to move to the new RHEL 6) wait for the feature to be sanctioned for use in Fedora by FPC 7) wait for packagers to start adopting the new thing ... 8) realize there's a flaw in X, hindering or even preventing its real-world usage 9) pray and beg for approval to fix it in RHEL 10) wait at least half a year for the next RHEL update to come out 11) go back to 7), rinse and repeat as necessary

For a practical example of the timescale of this "process" as things have been so far: opt-in install-time macro-expansion of scriptlets was implemented upstream in March 2010 and has been in Fedora since F15. Yet this relatively trivial thing *still* cannot be actually used in any version of Fedora, and nobody knows (or would not be allowed to say) when RHEL+1 comes out, so after 3.5 years from implementing the stupid thing, we're only at step 4) with an unknown (but in any case lengthy) time before being anywhere near 7).

So if people are still wondering why rpm moves at such glacial pace...

Having builders run Fedora latest-stable cuts the time from implementation to being deployable (or at least real-world testable) in Fedora literally by *years*, and the turn-around for steps 8-11 from months to days.

Let me say AWESOME one more time :)

Of course it does mean we could run into breakage if there's breakage in Fedora land, but thats good to know and fix too.

Yes, dogfooding on the builders can IMO only be a good thing for overall stability of Fedora, even if it might initially cause some extra hickups. Breaking the builders a couple of times and getting scolded for that ought to make people think a bit more about pushing potentially destabilizing updates :)

- Panu -

On 08/29/2013 11:47 AM, Vít Ondruch wrote:

Dne 29.8.2013 08:05, Panu Matilainen napsal(a):

...

For a practical example of the timescale of this "process" as things have been so far: opt-in install-time macro-expansion of scriptlets was implemented upstream in March 2010 and has been in Fedora since F15. Yet this relatively trivial thing *still* cannot be actually used in any version of Fedora, and nobody knows (or would not be allowed to say) when RHEL+1 comes out, so after 3.5 years from implementing the stupid thing, we're only at step 4) with an unknown (but in any case lengthy) time before being anywhere near 7).

Reading such great news, Panu, would you mind to prepare list of features we could start using in next Fedoras?

All in good time...

First we need to see this change actually go through all the way and stick. Secondly there isn't a whole lot of such features right now, as efforts have been concentrated on improving the existing, usable features rather than implementing new things because of the insane deployment "process". Finally, the use of such features within Fedora needs to be sanctioned by FPC and that's by no means guaranteed to happen (eg tilde-versioning was explicitly banned although it actually could already be used).

- Panu -

On 29 Aug 2013 18:12, "Kevin Fenzi" kevin@scrye.com wrote:

On Thu, 29 Aug 2013 09:05:24 +0300 Panu Matilainen pmatilai@laiskiainen.org wrote:

...snip...

...

Let me say AWESOME one more time :)

:)

I actually didn't know that this was so much of a bottleneck for you folks. ;(

Anyhow, glad we can move forward and hopefully get things rolling faster.

...

...

Of course it does mean we could run into breakage if there's breakage in Fedora land, but thats good to know and fix too.

Yes, dogfooding on the builders can IMO only be a good thing for overall stability of Fedora, even if it might initially cause some extra hickups. Breaking the builders a couple of times and getting scolded for that ought to make people think a bit more about pushing potentially destabilizing updates :)

Exactly.

I know we discovered a few issues in the last couple of years doing this with arm because of using the stable fedora arm release so I'm glad we're able to help :-)

Pete

kevin

-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct

I'm not saying this was a bad idea, but the Koji developers now have to test on both RHEL 6 and Fedora now. Let this message serve as a warning to them if they did not already know. ;)

They already were anyway due to the ARM builders in secondary so they've been getting reports from us for 3 years or so :-)

Peter