I was asked (with my Red Hat hat on) to put together a little report on areas in Fedora which could be improved with an investment in better automation. From what I'm working on myself, I'm aware of the gigantic need in the cloud images production process, and I've been keeping an eye on Tim Flink's autoqa revamp ideas. I expect there are others, because I know from my previous jobs that there's always a balance between building condiment-passing machines and just _passing the salt_. [1] Are there other things which could be made better if only someone came up with the spare time and resources to do the work?
On Tue, May 28, 2013 at 08:33:22AM -0400, Matthew Miller wrote:
I was asked (with my Red Hat hat on) to put together a little report on areas in Fedora which could be improved with an investment in better automation. From what I'm working on myself, I'm aware of the gigantic need in the cloud images production process, and I've been keeping an eye on Tim Flink's autoqa revamp ideas. I expect there are others, because I know from my previous jobs that there's always a balance between building condiment-passing machines and just _passing the salt_. [1] Are there other things which could be made better if only someone came up with the spare time and resources to do the work?
Hi!
Some observations/ideas:
1) The packager workflow is pretty tedious. There has been some improvement to it, but more can be done. Things like fedora-review and fedora-create-review (and bodhi!) are a huge help. But there are plenty of inefficient "blocking" points in the process.
For instance, once a new package is approved, only then does the submitter declare what branches they want with an scm admin request. They then wait for an scm admin to declare that they have created their branches, and then wait for a cronjob to run that gives them permission to push on those branches (manually). They then wait for their koji builds to finish to (manually) submit bodhi updates.
It would be nice if we could automate that whole process -- once a package is approved, if there were a "make-it-so" button that required no further intervention from the packager (but still required the keen eye of an scm admin).
There are further sequences down the pipeline like requesting that packages in testing be pushed to stable, but there are good arguments against automating that.
2) Continuous deployment for infrastructure. It has been tossed around in IRC, possibly at FUDCon as well. If application developers could "git push" on the develop branch and have those changes automatically roll out to our staging infrastructure -- that would save a lot of time. Packaging our apps, building rpms, signing them, copying them to our infra yum repos, rebuilding those repos, clearing the cache on the target machines, performing a yum update <-- that process is cumbersome.
I suspect that the "release only when we have accumulated enough changes to warrant enduring the burdensome release process" mode of deployment (as opposed to "release early, release often") also poses somewhat of a barrier to new contributors. They contribute a patch.. nice! When does it go live? When one of our overstretched sysadmin-mains can get around to it (it is required that one of them sign the package).
Caveat #2.1: There are some ways around this. Individuals can get around the requirement of having a sysadmin-main touch their test release by installing their rpm directly on the target machine. They still have to jump through some hoops to make it happen.
Caveat #2.2: This is one of the reasons we put so much work into our private cloud (dev nodes). There is no barrier there for teams to set up their own continuous deployment mechanism. This meets most needs, but we don't have a way to iterate rapidly on some of the more important pieces of our infrastructure. Apps/services that interact with each other don't quite work out on isolated cloud nodes. The bodhi masher? Koji? fedmsg? mirror manager? We can't necessarily test those on dev nodes (and some we can't test in staging -- resolving this down the road would save some headaches).
On Tue, 2013-05-28 at 09:45 -0400, Ralph Bean wrote:
The packager workflow is pretty tedious. There has been some improvement to it, but more can be done. Things like fedora-review and fedora-create-review (and bodhi!) are a huge help. But there are plenty of inefficient "blocking" points in the process.
For instance, once a new package is approved, only then does the submitter declare what branches they want with an scm admin request. They then wait for an scm admin to declare that they have created their branches, and then wait for a cronjob to run that gives them permission to push on those branches (manually). They then wait for their koji builds to finish to (manually) submit bodhi updates.
It would be nice if we could automate that whole process -- once a package is approved, if there were a "make-it-so" button that required no further intervention from the packager (but still required the keen eye of an scm admin).
There are further sequences down the pipeline like requesting that packages in testing be pushed to stable, but there are good arguments against automating that.
I've had the idea of having a service dedicated to package review for a while. It could be backed up with git so that the changes from the review directly end up in the git repository, tracking the whole history of the package since it was proposed for review. Of course it would also ask the submitter which branch he would like to have on his package and eventually fedora-review could be triggered for each new review created giving early feedback.
Apparently, I was trying to make something concrete of this idea 18months ago: http://ambre.pingoured.fr/cgit/review_srv.git/ But, although I still find the idea interesting, I have not pushed it further.
Pierre
----- Original Message -----
From: "Ralph Bean" rbean@redhat.com To: "Fedora Infrastructure" infrastructure@lists.fedoraproject.org Sent: Tuesday, May 28, 2013 6:45:43 AM Subject: Re: areas where we can invest in automation?
On Tue, May 28, 2013 at 08:33:22AM -0400, Matthew Miller wrote:
I was asked (with my Red Hat hat on) to put together a little report on areas in Fedora which could be improved with an investment in better automation. From what I'm working on myself, I'm aware of the gigantic need in the cloud images production process, and I've been keeping an eye on Tim Flink's autoqa revamp ideas. I expect there are others, because I know from my previous jobs that there's always a balance between building condiment-passing machines and just _passing the salt_. [1] Are there other things which could be made better if only someone came up with the spare time and resources to do the work?
Hi!
Some observations/ideas:
The packager workflow is pretty tedious. There has been some improvement to it, but more can be done. Things like fedora-review and fedora-create-review (and bodhi!) are a huge help. But there are plenty of inefficient "blocking" points in the process.
For instance, once a new package is approved, only then does the submitter declare what branches they want with an scm admin request. They then wait for an scm admin to declare that they have created their branches, and then wait for a cronjob to run that gives them permission to push on those branches (manually). They then wait for their koji builds to finish to (manually) submit bodhi updates.
It would be nice if we could automate that whole process -- once a package is approved, if there were a "make-it-so" button that required no further intervention from the packager (but still required the keen eye of an scm admin).
There are further sequences down the pipeline like requesting that packages in testing be pushed to stable, but there are good arguments against automating that.
Continuous deployment for infrastructure. It has been tossed around in IRC, possibly at FUDCon as well. If application developers could "git push" on the develop branch and have those changes automatically roll out to our staging infrastructure -- that would save a lot of time. Packaging our apps, building rpms, signing them, copying them to our infra yum repos, rebuilding those repos, clearing the cache on the target machines, performing a yum update <-- that process is cumbersome.
I suspect that the "release only when we have accumulated enough changes to warrant enduring the burdensome release process" mode of deployment (as opposed to "release early, release often") also poses somewhat of a barrier to new contributors. They contribute a patch.. nice! When does it go live? When one of our overstretched sysadmin-mains can get around to it (it is required that one of them sign the package).
This would be super. And yes - I think that the "instant gratification" combined with having to know what is likely a lot less process seems like it would yield additional contributions.
Do we have tiers for apps according to their "must stay-up" level - things that are critical vs. NTH? It seems like something where we could probably pick a non-critical app or two and work out what the process would be and basically take that process for a in-production-test-run.
Caveat #2.1: There are some ways around this. Individuals can get around the requirement of having a sysadmin-main touch their test release by installing their rpm directly on the target machine. They still have to jump through some hoops to make it happen.
Caveat #2.2: This is one of the reasons we put so much work into our private cloud (dev nodes). There is no barrier there for teams to set up their own continuous deployment mechanism. This meets most needs, but we don't have a way to iterate rapidly on some of the more important pieces of our infrastructure. Apps/services that interact with each other don't quite work out on isolated cloud nodes. The bodhi masher? Koji? fedmsg? mirror manager? We can't necessarily test those on dev nodes (and some we can't test in staging -- resolving this down the road would save some headaches).
So I'm definitely treading into "woefully technically unedumacated" territory - but could we not just take snapshots of the associated apps/services/databases, test whatever changes are incoming against the snapshots, and if it passes, push the changes to production and dump the tested nodes? (I realize this is simplifying things - we might have changes that require changes in both places, etc - but just wondering if I'm totally off-base here.) Or is this just a matter of "the infra things aren't running in the same place"?
-robyn
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Tue, 28 May 2013 09:45:43 -0400 Ralph Bean rbean@redhat.com wrote:
Some observations/ideas:
The packager workflow is pretty tedious. There has been some improvement to it, but more can be done. Things like fedora-review and fedora-create-review (and bodhi!) are a huge help. But there are plenty of inefficient "blocking" points in the process.
For instance, once a new package is approved, only then does the submitter declare what branches they want with an scm admin request. They then wait for an scm admin to declare that they have created their branches, and then wait for a cronjob to run that gives them permission to push on those branches (manually). They then wait for their koji builds to finish to (manually) submit bodhi updates.
It would be nice if we could automate that whole process -- once a package is approved, if there were a "make-it-so" button that required no further intervention from the packager (but still required the keen eye of an scm admin).
We've talked about this for ages. ;) It was going to be part of a new pkgdb, then standalone, then in a composedb thing. Not sure where it is now. ;)
The scm admin script currently in use is pretty easy... it's usually that we simply only have one scm admin at a time thats processing things. :(
There are further sequences down the pipeline like requesting that packages in testing be pushed to stable, but there are good arguments against automating that.
To some extent thats already automated with the karma system. ;)
- Continuous deployment for infrastructure. It has been tossed around in IRC, possibly at FUDCon as well. If application developers could "git push" on the develop branch and have those changes automatically roll out to our staging infrastructure -- that would save a lot of time. Packaging our apps, building rpms, signing them, copying them to our infra yum repos, rebuilding those repos, clearing the cache on the target machines, performing a yum update <-- that process is cumbersome.
yeah, but I think of that as more a dev thing than staging. Staging is something you keep very close to production and only right before you intend to go to production do you update staging.
Others might have differing thoughts on that...
I suspect that the "release only when we have accumulated enough changes to warrant enduring the burdensome release process" mode of deployment (as opposed to "release early, release often") also poses somewhat of a barrier to new contributors. They contribute a patch.. nice! When does it go live? When one of our overstretched sysadmin-mains can get around to it (it is required that one of them sign the package).
Yeah.
Caveat #2.1: There are some ways around this. Individuals can get around the requirement of having a sysadmin-main touch their test release by installing their rpm directly on the target machine. They still have to jump through some hoops to make it happen.
Caveat #2.2: This is one of the reasons we put so much work into our private cloud (dev nodes). There is no barrier there for teams to set up their own continuous deployment mechanism. This meets most needs, but we don't have a way to iterate rapidly on some of the more important pieces of our infrastructure. Apps/services that interact with each other don't quite work out on isolated cloud nodes. The bodhi masher? Koji? fedmsg? mirror manager? We can't necessarily test those on dev nodes (and some we can't test in staging -- resolving this down the road would save some headaches).
Yeah, something to ponder on for sure.
kevin
On Tue, May 28, 2013 at 08:33:22 -0400, Matthew Miller mattdm@fedoraproject.org wrote:
I was asked (with my Red Hat hat on) to put together a little report on areas in Fedora which could be improved with an investment in better automation. From what I'm working on myself, I'm aware of the gigantic need in the cloud images production process, and I've been keeping an eye on Tim Flink's autoqa revamp ideas. I expect there are others, because I know from my previous jobs that there's always a balance between building condiment-passing machines and just _passing the salt_. [1] Are there other things which could be made better if only someone came up with the spare time and resources to do the work?
If we are willing to have packages being signed, just mean they were built with koji, we could have a tool that could doing the signing so that packages could move from pending to testing or updates without human intervention and with the same key being used for all koji (non-scratch) builds, hard links could be used to save space on the mirrors for packages that appear in multiple repos.
On Tue, 28 May 2013 08:49:03 -0500 Bruno Wolff III bruno@wolff.to wrote:
If we are willing to have packages being signed, just mean they were built with koji, we could have a tool that could doing the signing so that packages could move from pending to testing or updates without human intervention and with the same key being used for all koji (non-scratch) builds, hard links could be used to save space on the mirrors for packages that appear in multiple repos.
I looked into this earlier this year. There is a koji plugin to do this, but it's very unacceptable to koji upstream. It also means we need to keep signed packages for everything for all time, which is more space on our end and finally it's a nasty security issue as the password for the key has to be stored in the plugin that does this.
A possibly nicer idea would be moving to signed repodata. Then if you see the correct sha512 or whatever in the signed repodata you know the package is good. That does have downsides too tho, as you can't verify things once they are obsolete or off line.
Its not an easy problem to solve...
kevin
Off the top of my head:
- I would LOVE to have the nightly-compose stuff automated. I just never get time to work on it. Ideally it would just be part of rawhide or branched compose scripts, run the builds, generate a nice web page that has FAILED or OK or whatever on it, as well as sending a report to the devel/test lists with things like packages added, packages removed, size change, etc.
- For that matter I know Dennis has wanted a compose app for a long time that would be better than our current cron jobs to compose branched/rawhide/testcomposes/release candidates. Such an app could continue to provide email reports, but could also have a dashboard so you could see what was what and view logs easily, etc.
- Longer term/down the road, it would be nice if koji had cloud awareness. Instead of having a static pool of builders preallocated, if it could just talk to a openstack cloud or something and ask for a instance, wait for it, use it, terminate it.
Kevin
Dne 28.5.2013 20:08, Kevin Fenzi napsal(a):
Off the top of my head:
- Longer term/down the road, it would be nice if koji had cloud awareness. Instead of having a static pool of builders preallocated, if it could just talk to a openstack cloud or something and ask for a instance, wait for it, use it, terminate it.
I proposed this to Koji maintainers a while ago. Koji should share similar BE with Copr IMO, which does this already. But unfortunately, this idea was promptly dismissed, since sharing on Mock level is enough in their opinion :/
Vít
On 05/28/2013 08:33 AM, Matthew Miller wrote:
I was asked (with my Red Hat hat on) to put together a little report on areas in Fedora which could be improved with an investment in better automation.
I just posted some suggestions to
https://lists.fedoraproject.org/pipermail/devel/2013-May/183459.html
Rahul
infrastructure@lists.fedoraproject.org
-
Bruno Wolff III -
Kevin Fenzi -
Matthew Miller -
Pierre-Yves Chibon -
Rahul Sundaram -
Ralph Bean -
Robyn Bergeron -
Vít Ondruch