Background: appliance-creator, as part of Thincrust, isn't really maintained anymore. It'd be nice to switch to something else. There's two main options that I can see:
1. ami-creator, as maintained by the Eucalyptus project. This basically works the same way as the appliance-creator but is maintained.
2. livemedia-creator. This does a "real" install, using anaconda inside a VM.
The second has the obvious disadvantage of needing to launch a VM. Richard Jones suggest that running under fully-emulated qemu layered in a VM builder might not be so bad because we'll be io-bound anyway.
It has the advantage of doing exactly what anaconda does, which will make it easier to track the standard Fedora install. We'd also be less insulated from possible breakage due to anaconda development, which from an overall point of view is both good and bad.
In any case, I'd like to start running an image creation task automatically nightly if possible.
What do the people actually running the builders think about this?
On Mon, 12 Nov 2012, Matthew Miller wrote:
Background: appliance-creator, as part of Thincrust, isn't really maintained anymore. It'd be nice to switch to something else. There's two main options that I can see:
- ami-creator, as maintained by the Eucalyptus project. This basically
works the same way as the appliance-creator but is maintained.
- livemedia-creator. This does a "real" install, using anaconda inside a
VM.
The second has the obvious disadvantage of needing to launch a VM. Richard Jones suggest that running under fully-emulated qemu layered in a VM builder might not be so bad because we'll be io-bound anyway.
It has the advantage of doing exactly what anaconda does, which will make it easier to track the standard Fedora install. We'd also be less insulated from possible breakage due to anaconda development, which from an overall point of view is both good and bad.
In any case, I'd like to start running an image creation task automatically nightly if possible.
What do the people actually running the builders think about this?
Dumb question time - for building an image using ami-creator you just need fast access to the just-built/koji repos an instance to run ami-creator on and a place to store the results....
Why don't we spin up a persistent euca instance, give you some disk space and a cron job. You can just run ami-creator w/a sensible kickstart...
Trivial and disposable.
-sv
On Mon, 12 Nov 2012 10:48:18 -0500 (EST) Seth Vidal skvidal@fedoraproject.org wrote:
On Mon, 12 Nov 2012, Matthew Miller wrote:
Background: appliance-creator, as part of Thincrust, isn't really maintained anymore. It'd be nice to switch to something else. There's two main options that I can see:
- ami-creator, as maintained by the Eucalyptus project. This
basically works the same way as the appliance-creator but is maintained.
- livemedia-creator. This does a "real" install, using anaconda
inside a VM.
The second has the obvious disadvantage of needing to launch a VM. Richard Jones suggest that running under fully-emulated qemu layered in a VM builder might not be so bad because we'll be io-bound anyway.
It has the advantage of doing exactly what anaconda does, which will make it easier to track the standard Fedora install. We'd also be less insulated from possible breakage due to anaconda development, which from an overall point of view is both good and bad.
In any case, I'd like to start running an image creation task automatically nightly if possible.
What do the people actually running the builders think about this?
Dumb question time - for building an image using ami-creator you just need fast access to the just-built/koji repos an instance to run ami-creator on and a place to store the results....
Why don't we spin up a persistent euca instance, give you some disk space and a cron job. You can just run ami-creator w/a sensible kickstart...
Trivial and disposable.
For short term/testing I think this is a great idea.
Longer term, we really need to figure out what "official" cloud images are going to be made with and where. I guess thats a conversation with koji upstream and rel-eng and cloud sig. I personally don't have any horse in that race, but having something thats controlled and repeatable is I am sure a must.
kevin
On Mon, 12 Nov 2012, Kevin Fenzi wrote:
Trivial and disposable.
For short term/testing I think this is a great idea.
Longer term, we really need to figure out what "official" cloud images are going to be made with and where. I guess thats a conversation with koji upstream and rel-eng and cloud sig. I personally don't have any horse in that race, but having something thats controlled and repeatable is I am sure a must.
2 things here: 1. I suspect the cloud architect can help us describe what "official" is :) 2. I see nothing which suggests we can repeat/replay koji appliance creation over time. We might be able to do it w/i one week - but much outside of that and I suspect rawhide will have drifted too much.
-sv
On Mon, Nov 12, 2012 at 10:48:18AM -0500, Seth Vidal wrote:
Why don't we spin up a persistent euca instance, give you some disk space and a cron job. You can just run ami-creator w/a sensible kickstart... Trivial and disposable.
That would be fine, as long as we can be very sure about the access control and chain of identity assurance.
I don't know the existing internal systems for this, but it'd be great to have auditable assurance that images which appear on the mirrors were built in a known-clean environment against the official repositories from a certain kickstart file in git.
For the purposes of Rawhide nightlies (pushed to alt.fedoraproject.org?), I'm perfectly fine with trusting me to do the right thing. :) For the alpha, beta, and final builds, as well as possible mid-release image updates, I'd like access to go through some control system, whatever that might be.
On Mon, 12 Nov 2012, Matthew Miller wrote:
On Mon, Nov 12, 2012 at 10:48:18AM -0500, Seth Vidal wrote:
Why don't we spin up a persistent euca instance, give you some disk space and a cron job. You can just run ami-creator w/a sensible kickstart... Trivial and disposable.
That would be fine, as long as we can be very sure about the access control and chain of identity assurance.
I don't know how/if we do that _now_.
I mean - we know who setup the koji builders and where koji lives but 'chain of identity assurance'..... That sounds awfully specific and legalistic.
Maybe you can describe what that phrase means to you so I can understand that a bit better.
I don't know the existing internal systems for this, but it'd be great to have auditable assurance that images which appear on the mirrors were built in a known-clean environment against the official repositories from a certain kickstart file in git.
'known clean'? What kind of clean do you want? If we use a new instance to build a new image is that clean enough? I'm certain I could do all of it in a single ansible playbook: spin up a new instance in euca, attach a set of disks, run ami-creator, retrieve the results. It's not very difficult at all, actually.
For the purposes of Rawhide nightlies (pushed to alt.fedoraproject.org?), I'm perfectly fine with trusting me to do the right thing. :) For the alpha, beta, and final builds, as well as possible mid-release image updates, I'd like access to go through some control system, whatever that might be.
Let me know what you think the control system requires to be considered "safe".
-sv
On Mon, Nov 12, 2012 at 12:08:35PM -0500, Seth Vidal wrote:
That would be fine, as long as we can be very sure about the access control and chain of identity assurance.
I don't know how/if we do that _now_. I mean - we know who setup the koji builders and where koji lives but 'chain of identity assurance'..... That sounds awfully specific and legalistic. Maybe you can describe what that phrase means to you so I can understand that a bit better.
Sure. :)
First, I'll happily assume that people who have a admin access to the builders are black box of trust -- there's plenty of internal accountability and logging and whatever.
What I mean is: when I build a package for Fedora, I go through the Koji build system. I can't just kludge up a binary RPM and have it get sent out into the mirror. And, anyone can go into Koji and see the packages I've built -- and see how they were built, if they want. And although the GPG package signing process is also a black box to some degree, Bodhi gives pretty good transparency into the path an update takes.
This isn't just good for distribution security, but it's also good for repeatability, and it's convenient for me as a packager. I want all of that for the cloud images. Any (technically-minded) end user should be able to work back from the image checksum to the actual build logs.
'known clean'? What kind of clean do you want? If we use a new instance to build a new image is that clean enough? I'm certain I could do all of it in a single ansible playbook: spin up a new instance in euca, attach a set of disks, run ami-creator, retrieve the results. It's not very difficult at all, actually.
Yes that kind of clean.
For the purposes of Rawhide nightlies (pushed to alt.fedoraproject.org?), I'm perfectly fine with trusting me to do the right thing. :) For the alpha, beta, and final builds, as well as possible mid-release image updates, I'd like access to go through some control system, whatever that might be.
Let me know what you think the control system requires to be considered "safe".
I think:
* all input to the build command should be recorded - save exact command-line arguments - either collect the kickstart file provided or require it to be pulled from a Fedora hosted git repo (and record what was pulled)
* checksums of the output should be generated and recorded
* no opportunity for the person doing the build to write anything into the build process while it's in progress, and no opportunity to alter the above records.
does that seem sound? Also build logs should be stored, but that's more for information, debugging, and convenience than anything else.
If the person doing the build normally has admin access on Fedora build systems, they shouldn't on this one. Trust but verify and all that.
What I mean is: when I build a package for Fedora, I go through the Koji build system. I can't just kludge up a binary RPM and have it get sent out into the mirror. And, anyone can go into Koji and see the packages I've built -- and see how they were built, if they want. And although the GPG package signing process is also a black box to some degree, Bodhi gives pretty good transparency into the path an update takes.
They can see what happened, they may not actually be able to get the pkg... We don't keep pkgs forever.
This isn't just good for distribution security, but it's also good for repeatability, and it's convenient for me as a packager. I want all of that for the cloud images. Any (technically-minded) end user should be able to work back from the image checksum to the actual build logs.
For purposes of repeatability it seems like you just need: 1. the pkgs+checksums on the system you are running ami-creator from 2. the pkgs+checksums used in the ami-creation itself. 3. the set of pkgs+checksums available at creation time
All of the above mock stores now.
'known clean'? What kind of clean do you want? If we use a new instance to build a new image is that clean enough? I'm certain I could do all of it in a single ansible playbook: spin up a new instance in euca, attach a set of disks, run ami-creator, retrieve the results. It's not very difficult at all, actually.
Yes that kind of clean.
This is, ultimately, the same process we're using for spinning up build instances for coprs.
After this week, hopefully, if you want some help setting this up for your ami-creator runs I can help. It's really not difficult at all anymore.
I think:
- all input to the build command should be recorded
- save exact command-line arguments
- wouldn't be any command line arguments at all, I hope.
- either collect the kickstart file provided or require it to be pulled
- exactly - also I'm not sure why it should NOT go into the created img itself in /root or some other path.
- checksums of the output should be generated and recorded
not sure where you want to record them - but that's a detail.
- no opportunity for the person doing the build to write anything into the
build process while it's in progress, and no opportunity to alter the above records.
There's always an opportunity. If you can spin up an instance you can also get to that instance. Otherwise you wouldn't be able to run ami-creator sensibly.
But whereever we decide to draw the line - storing the state information is not very hard (or even very large)
does that seem sound? Also build logs should be stored, but that's more for information, debugging, and convenience than anything else.
again - stored where?
If the person doing the build normally has admin access on Fedora build systems, they shouldn't on this one. Trust but verify and all that.
this criteria may be tricky... There's just not that many of us.
-sv
What I mean is: when I build a package for Fedora, I go through the Koji build system. I can't just kludge up a binary RPM and have it get sent out into the mirror. And, anyone can go into Koji and see the packages I've built -- and see how they were built, if they want. And although the GPG package signing process is also a black box to some degree, Bodhi gives pretty good transparency into the path an update takes.
They can see what happened, they may not actually be able to get the pkg... We don't keep pkgs forever.
This isn't just good for distribution security, but it's also good for repeatability, and it's convenient for me as a packager. I want all of that for the cloud images. Any (technically-minded) end user should be able to work back from the image checksum to the actual build logs.
For purposes of repeatability it seems like you just need:
- the pkgs+checksums on the system you are running ami-creator from
- the pkgs+checksums used in the ami-creation itself.
- the set of pkgs+checksums available at creation time
All of the above mock stores now.
'known clean'? What kind of clean do you want? If we use a new instance to build a new image is that clean enough? I'm certain I could do all of it in a single ansible playbook: spin up a new instance in euca, attach a set of disks, run ami-creator, retrieve the results. It's not very difficult at all, actually.
Yes that kind of clean.
This is, ultimately, the same process we're using for spinning up build instances for coprs.
After this week, hopefully, if you want some help setting this up for your ami-creator runs I can help. It's really not difficult at all anymore.
I think:
- all input to the build command should be recorded
- save exact command-line arguments
- wouldn't be any command line arguments at all, I hope.
- either collect the kickstart file provided or require it to be pulled
- exactly - also I'm not sure why it should NOT go into the createdimg itself in /root or some other path.
- checksums of the output should be generated and recorded
not sure where you want to record them - but that's a detail.
- no opportunity for the person doing the build to write anything into
the build process while it's in progress, and no opportunity to alter the above records.
There's always an opportunity. If you can spin up an instance you can also get to that instance. Otherwise you wouldn't be able to run ami-creator sensibly.
But whereever we decide to draw the line - storing the state information is not very hard (or even very large)
does that seem sound? Also build logs should be stored, but that's more for information, debugging, and convenience than anything else.
again - stored where?
If the person doing the build normally has admin access on Fedora build systems, they shouldn't on this one. Trust but verify and all that.
this criteria may be tricky... There's just not that many of us.
Plus a lot of the releng people like kevin and dgilmore are in sysadmin-main which has admin access everywhere.
-sv
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Mon, Nov 12, 2012 at 08:22:17PM -0600, Nick Bebout wrote:
this criteria may be tricky... There's just not that many of us.
Plus a lot of the releng people like kevin and dgilmore are in sysadmin-main which has admin access everywhere.
Worrying about this kind of thing is officially outside my scope, but I really think it's good practice. Even when we have awesome people we really know we can trust.
That said, it is probably reasonable to have different access methods, so that users *could* theoretically access the backend system, they'd at the very least do so through different channels and when intentionally wearing a different (mental) hat.
On Mon, 12 Nov 2012 21:41:59 -0500 Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Nov 12, 2012 at 08:22:17PM -0600, Nick Bebout wrote:
this criteria may be tricky... There's just not that many of us.
Plus a lot of the releng people like kevin and dgilmore are in sysadmin-main which has admin access everywhere.
Worrying about this kind of thing is officially outside my scope, but I really think it's good practice. Even when we have awesome people we really know we can trust.
Sure. Problem is we need some people to maintain the system, so they need some level of access to do that, apply updates, etc.
That said, things like updates or sudo calls or the like are logged and watched (at least I do look at all the sudo's and logins)
I think we are getting a bit in the weeds here tho...
That said, it is probably reasonable to have different access methods, so that users *could* theoretically access the backend system, they'd at the very least do so through different channels and when intentionally wearing a different (mental) hat.
Right.
Ideally the process logs and saves off everything it does, and if there needs to be a backend change thats logged and audited as well.
Trust, but verify.
Really the things koji has going for it in this is that it has a database thats a easy place to store and view audit logs for things like this. So you can say "hey, the f18 beta image, how was it made" and can easily find the logs and look thought them. We could easily setup something that does something similar with a cloud instance, just needs more tooling.
kevin
On 2012-11-12 12:38, Seth Vidal wrote:
What I mean is: when I build a package for Fedora, I go through the Koji build system. I can't just kludge up a binary RPM and have it get sent out into the mirror. And, anyone can go into Koji and see the packages I've built -- and see how they were built, if they want. And although the GPG package signing process is also a black box to some degree, Bodhi gives pretty good transparency into the path an update takes.
They can see what happened, they may not actually be able to get the pkg... We don't keep pkgs forever.
But we do for releases, from which all previous release images have been spun, AFAIK. The cloud SIG made sure of that quite early on in its lifetime to keep everything GPL-compliant, and it might be worth keeping in mind as we consider how to build images for future releases.
-- Garrett Holmstrom
On Tue, 13 Nov 2012, Garrett Holmstrom wrote:
On 2012-11-12 12:38, Seth Vidal wrote:
What I mean is: when I build a package for Fedora, I go through the Koji build system. I can't just kludge up a binary RPM and have it get sent out into the mirror. And, anyone can go into Koji and see the packages I've built -- and see how they were built, if they want. And although the GPG package signing process is also a black box to some degree, Bodhi gives pretty good transparency into the path an update takes.
They can see what happened, they may not actually be able to get the pkg... We don't keep pkgs forever.
But we do for releases, from which all previous release images have been spun, AFAIK. The cloud SIG made sure of that quite early on in its lifetime to keep everything GPL-compliant, and it might be worth keeping in mind as we consider how to build images for future releases.
and I a pretty sure for releases we're talking about something else. This is for continual/nightly image generation, unless I'm mistaken.
-sv
On Wed, Nov 14, 2012 at 09:31:44AM -0500, Seth Vidal wrote:
But we do for releases, from which all previous release images have been spun, AFAIK. The cloud SIG made sure of that quite early on in its lifetime to keep everything GPL-compliant, and it might be worth keeping in mind as we consider how to build images for future releases.
and I a pretty sure for releases we're talking about something else. This is for continual/nightly image generation, unless I'm mistaken.
I was thinking we'd generate the releases in the same way, but am willing to be disabused of that notion.
On Wed, 14 Nov 2012, Matthew Miller wrote:
On Wed, Nov 14, 2012 at 09:31:44AM -0500, Seth Vidal wrote:
But we do for releases, from which all previous release images have been spun, AFAIK. The cloud SIG made sure of that quite early on in its lifetime to keep everything GPL-compliant, and it might be worth keeping in mind as we consider how to build images for future releases.
and I a pretty sure for releases we're talking about something else. This is for continual/nightly image generation, unless I'm mistaken.
I was thinking we'd generate the releases in the same way, but am willing to be disabused of that notion.
Same software for image generation, certainly. Not obvious to me that generating them in the same place is worthwhile.
-sv
On Wed, 14 Nov 2012 09:49:52 -0500 (EST) Seth Vidal skvidal@fedoraproject.org wrote:
On Wed, 14 Nov 2012, Matthew Miller wrote:
On Wed, Nov 14, 2012 at 09:31:44AM -0500, Seth Vidal wrote:
But we do for releases, from which all previous release images have been spun, AFAIK. The cloud SIG made sure of that quite early on in its lifetime to keep everything GPL-compliant, and it might be worth keeping in mind as we consider how to build images for future releases.
and I a pretty sure for releases we're talking about something else. This is for continual/nightly image generation, unless I'm mistaken.
I was thinking we'd generate the releases in the same way, but am willing to be disabused of that notion.
Same software for image generation, certainly. Not obvious to me that generating them in the same place is worthwhile.
As long as the env is similar enough I don't know that it matters. Ideally it would be very similar tho, as you don't want your perfectly working and nice nightly images to fail to work when "officially" composed.
kevin
The key to license compliance is to post the SRPMs corresponding to the bits in the image, concurrent with the image, and on the same lifetime, so people have the opportunity (not the requirement) to download the SRPMs when they download the image.
If they choose not to download the SRPMs at that time, and you delete both the image and SRPMs concurrently, then the user is out of luck if they later decide they want the SRPMs, but it's not a violation on your part.
-- Matt Domsch Technology Strategist Dell | Office of the CTO ________________________________________ From: infrastructure-bounces@lists.fedoraproject.org [infrastructure-bounces@lists.fedoraproject.org] On Behalf Of Matthew Miller [mattdm@fedoraproject.org] Sent: Wednesday, November 14, 2012 8:37 AM To: Fedora Infrastructure Subject: Re: tools for building cloud images in the buildsystem
On Wed, Nov 14, 2012 at 09:31:44AM -0500, Seth Vidal wrote:
But we do for releases, from which all previous release images have been spun, AFAIK. The cloud SIG made sure of that quite early on in its lifetime to keep everything GPL-compliant, and it might be worth keeping in mind as we consider how to build images for future releases.
and I a pretty sure for releases we're talking about something else. This is for continual/nightly image generation, unless I'm mistaken.
I was thinking we'd generate the releases in the same way, but am willing to be disabused of that notion.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Mon, Nov 12, 2012 at 10:16 AM, Matthew Miller mattdm@fedoraproject.org wrote:
Background: appliance-creator, as part of Thincrust, isn't really maintained anymore. It'd be nice to switch to something else. There's two main options that I can see:
- ami-creator, as maintained by the Eucalyptus project. This basically works the same way as the appliance-creator but is maintained.
To be fair, the original maintainer, Jeremy Katz, is actually picking up the ami-creator project again. He's taken a few of my patches, and Eucalyptus will probably start using his version soon. We'll be contributors, but he's the maintainer. :)
Andy
- livemedia-creator. This does a "real" install, using anaconda inside a VM.
The second has the obvious disadvantage of needing to launch a VM. Richard Jones suggest that running under fully-emulated qemu layered in a VM builder might not be so bad because we'll be io-bound anyway.
It has the advantage of doing exactly what anaconda does, which will make it easier to track the standard Fedora install. We'd also be less insulated from possible breakage due to anaconda development, which from an overall point of view is both good and bad.
In any case, I'd like to start running an image creation task automatically nightly if possible.
What do the people actually running the builders think about this?
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
infrastructure@lists.fedoraproject.org
-
Andy Grimm -
Garrett Holmstrom -
Kevin Fenzi -
Matt Domsch -
Matthew Miller -
Nick Bebout -
Seth Vidal