Hi,
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there if possible.
--Krishna
On Thu, May 09, 2013 at 04:18:45PM -0700, Krishna Raman wrote:
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there if possible.
There were some testing ones but we didn't announce an official one, which we do plan to do for the beta.
Can I get a hold of the test image?
-kr On May 10, 2013 8:20 AM, "Matthew Miller" mattdm@fedoraproject.org wrote:
On Thu, May 09, 2013 at 04:18:45PM -0700, Krishna Raman wrote:
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there if
possible.
There were some testing ones but we didn't announce an official one, which we do plan to do for the beta.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ < mattdm@fedoraproject.org> _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
On Fri, May 10, 2013 at 5:22 PM, Krishna Raman kraman@gmail.com wrote:
Can I get a hold of the test image?
I'd be interested too.
...Juerg
-kr
On May 10, 2013 8:20 AM, "Matthew Miller" mattdm@fedoraproject.org wrote:
On Thu, May 09, 2013 at 04:18:45PM -0700, Krishna Raman wrote:
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there if possible.
There were some testing ones but we didn't announce an official one, which we do plan to do for the beta.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
You can build them yourself: https://git.fedorahosted.org/cgit/cloud-kickstarts.git/
On Fri, May 10, 2013 at 12:36 PM, Juerg Haefliger juergh@gmail.com wrote:
On Fri, May 10, 2013 at 5:22 PM, Krishna Raman kraman@gmail.com wrote:
Can I get a hold of the test image?
I'd be interested too.
...Juerg
-kr
On May 10, 2013 8:20 AM, "Matthew Miller" mattdm@fedoraproject.org
wrote:
On Thu, May 09, 2013 at 04:18:45PM -0700, Krishna Raman wrote:
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there
if
possible.
There were some testing ones but we didn't announce an official one,
which
we do plan to do for the beta.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
On Fri, May 10, 2013 at 3:17 PM, Ricardo Arguello < ricardo.arguello@gmail.com> wrote:
You can build them yourself: https://git.fedorahosted.org/cgit/cloud-kickstarts.git/
Using these commands:
# appliance-creator --name f19-x86_64 --config=fedora-19-x86_64-ec2.ks # qemu-img convert -f raw -c -O qcow2 f19-x86_64-sda.raw f19-x86_64-sda.qcow2
On Fri, May 10, 2013 at 12:36 PM, Juerg Haefliger juergh@gmail.com wrote:
On Fri, May 10, 2013 at 5:22 PM, Krishna Raman kraman@gmail.com wrote:
Can I get a hold of the test image?
I'd be interested too.
...Juerg
-kr
On May 10, 2013 8:20 AM, "Matthew Miller" mattdm@fedoraproject.org
wrote:
On Thu, May 09, 2013 at 04:18:45PM -0700, Krishna Raman wrote:
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there
if
possible.
There were some testing ones but we didn't announce an official one,
which
we do plan to do for the beta.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
On May 10, 2013, at 2:48 PM, Ricardo Arguello ricardo.arguello@gmail.com wrote:
On Fri, May 10, 2013 at 3:17 PM, Ricardo Arguello ricardo.arguello@gmail.com wrote: You can build them yourself: https://git.fedorahosted.org/cgit/cloud-kickstarts.git/
Using these commands:
# appliance-creator --name f19-x86_64 --config=fedora-19-x86_64-ec2.ks # qemu-img convert -f raw -c -O qcow2 f19-x86_64-sda.raw f19-x86_64-sda.qcow2
Thanks for the steps but I am still lost.
What do I do once I have the qcow2 image? Can you point me to the location of docs on Fedora site which will help me get started on ec2?
I only see a mention of BoxGrinder which doesnt seem to support anything beyond F16.
Thanks --kr
On Fri, May 10, 2013 at 12:36 PM, Juerg Haefliger juergh@gmail.com wrote: On Fri, May 10, 2013 at 5:22 PM, Krishna Raman kraman@gmail.com wrote:
Can I get a hold of the test image?
I'd be interested too.
...Juerg
-kr
On May 10, 2013 8:20 AM, "Matthew Miller" mattdm@fedoraproject.org wrote:
On Thu, May 09, 2013 at 04:18:45PM -0700, Krishna Raman wrote:
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there if possible.
There were some testing ones but we didn't announce an official one, which we do plan to do for the beta.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
I made an effort to get a Fedora 19 image running in EC2 this week, and in the end, I undid a couple of the recent changes to make it work for me. Here's a brief overview of what I did:
1) appliance-creator was giving me an error when the "part" line of the kickstart had no "ondisk" option, so I added "--ondisk xvda". This shouldn't be needed, but it helped in my case. 2) I also commented out the "Zeroing out empty space" postinstall stuff, because it drastically increases the image build time for not much benefit, IMHO. 3) To build the image, I ran "appliance-creator --name f19-x86_64 --config=cloud-kickstarts/generic/fedora-19-x86_64-cloud.ks" 4) I compressed the image for transport to ec2: tar -cSzf f19.tgz f19-x86_64/f19-x86_64-xvda.raw 5) I launched an F18 instance and attached a new 10 GB volume to it 6) I copied the tarball (which was only about 240 MB) to the instance and extracted with "tar xSzf f19.tgz" to preserve sparseness. 7) I tried dumping the whole disk onto the volume, but I had issues with that, so in the end I just dumped the filesystem into the volume instead:
losetup -f --show f19-x86_64/f19-x86_64-xvda.raw kpartx -a /dev/loop0 dd if=/dev/mapper/loop0p1 bs=1M of=/dev/xvdf
8) Because I was using the filesystem instead of the whole disk, I had to change "(hd0,0)" to "(hd0)" in grub.conf
After some failed boot attempts, I did a couple of other things:
9) I removed the "splashimage" line from grub.conf, because it's certainly not useful and could be harmful 10) I copied /boot/grub/grub.conf to /boot/grub/menu.lst, because I don't know whether pvgrub in EC2 reliably reads both files (it's supposed to).
And finally, I mounted the volume, took a snapshot, and registered with:
euca-register -s snap-95070fcf --kernel aki-825ea7eb -d "Fedora 19 Alpha" -n "fedora19-20130515.2" --architecture x86_64
I'm sure I did some extra steps here, and that this is not exactly how Matt and Dennis intend for images to be deployed, but absent any detailed instructions from them, I hope this will prove to be a useful starting point for people. Matt and Dennis, feel free to tell us about everything I've done wrong here. :-)
Andy
On Mon, May 13, 2013 at 2:57 PM, Krishna Raman kraman@gmail.com wrote:
On May 10, 2013, at 2:48 PM, Ricardo Arguello ricardo.arguello@gmail.com wrote:
On Fri, May 10, 2013 at 3:17 PM, Ricardo Arguello < ricardo.arguello@gmail.com> wrote:
You can build them yourself: https://git.fedorahosted.org/cgit/cloud-kickstarts.git/
Using these commands:
# appliance-creator --name f19-x86_64 --config=fedora-19-x86_64-ec2.ks # qemu-img convert -f raw -c -O qcow2 f19-x86_64-sda.raw f19-x86_64-sda.qcow2
Thanks for the steps but I am still lost.
What do I do once I have the qcow2 image? Can you point me to the location of docs on Fedora site which will help me get started on ec2?
I only see a mention of BoxGrinder which doesnt seem to support anything beyond F16.
Thanks --kr
On Fri, May 10, 2013 at 12:36 PM, Juerg Haefliger juergh@gmail.comwrote:
On Fri, May 10, 2013 at 5:22 PM, Krishna Raman kraman@gmail.com wrote:
Can I get a hold of the test image?
I'd be interested too.
...Juerg
-kr
On May 10, 2013 8:20 AM, "Matthew Miller" mattdm@fedoraproject.org
wrote:
On Thu, May 09, 2013 at 04:18:45PM -0700, Krishna Raman wrote:
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there
if
possible.
There were some testing ones but we didn't announce an official one,
which
we do plan to do for the beta.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
On Wed, May 15, 2013 at 10:53:44AM -0400, Andy Grimm wrote:
I made an effort to get a Fedora 19 image running in EC2 this week, and in the end, I undid a couple of the recent changes to make it work for me. Here's a brief overview of what I did:
- appliance-creator was giving me an error when the "part" line of the
kickstart had no "ondisk" option, so I added "--ondisk xvda". This shouldn't be needed, but it helped in my case.
I have a patch to appliance-creator which _should_ make this optional -- Dennis is going to take over upstream maintenance and I think will include that.
Is "xvda" right for _within_ appliance-creator?
- I also commented out the "Zeroing out empty space" postinstall stuff,
because it drastically increases the image build time for not much benefit, IMHO.
One time image build cost vs. whatever benefit multipled by every time the image is used. :)
- I removed the "splashimage" line from grub.conf, because it's certainly
not useful and could be harmful
https://bugzilla.redhat.com/show_bug.cgi?id=963294
- I copied /boot/grub/grub.conf to /boot/grub/menu.lst, because I don't
know whether pvgrub in EC2 reliably reads both files (it's supposed to).
I think it'll be okay either way, but maybe we should make a symlink?
On Wed, May 15, 2013 at 11:18:04AM -0400, Matthew Miller wrote:
- I also commented out the "Zeroing out empty space" postinstall stuff,
because it drastically increases the image build time for not much benefit, IMHO.
One time image build cost vs. whatever benefit multipled by every time the image is used. :)
To put some numbers behind it, the compressed qcow2 image with the dd to zero empty space is 215M out of appliance-creator. Without it, it's 242M.
If we use a smaller blocksize, or an incrementally shrinking one, we can shrink it by another megabyte, at the cost of even more build time. I think that's probably across the worth-it threshold, though.
Using virt-sparsify seems to take even longer, but does get it down to 208M. That requires libguestfs, which we can't do in the current build system, but will be able to in the future one. (Feature rescheduled for F20.)
Matthew Miller (mattdm@fedoraproject.org) said:
On Wed, May 15, 2013 at 11:18:04AM -0400, Matthew Miller wrote:
- I also commented out the "Zeroing out empty space" postinstall stuff,
because it drastically increases the image build time for not much benefit, IMHO.
One time image build cost vs. whatever benefit multipled by every time the image is used. :)
To put some numbers behind it, the compressed qcow2 image with the dd to zero empty space is 215M out of appliance-creator. Without it, it's 242M.
If we use a smaller blocksize, or an incrementally shrinking one, we can shrink it by another megabyte, at the cost of even more build time. I think that's probably across the worth-it threshold, though.
Using virt-sparsify seems to take even longer, but does get it down to 208M. That requires libguestfs, which we can't do in the current build system, but will be able to in the future one. (Feature rescheduled for F20.)
Why not use fstrim on the image file?
Bill
On Tue, May 21, 2013 at 05:39:21PM -0400, Bill Nottingham wrote:
Using virt-sparsify seems to take even longer, but does get it down to 208M. That requires libguestfs, which we can't do in the current build system, but will be able to in the future one. (Feature rescheduled for F20.)
Why not use fstrim on the image file?
Um. Because no good reason? This is both fast and produces the smallest size.
There was some problem with it months ago, and I had discarded it as an option. But clearly that was wrong!
On Tue, May 21, 2013 at 06:23:13PM -0400, Matthew Miller wrote:
Using virt-sparsify seems to take even longer, but does get it down to 208M. That requires libguestfs, which we can't do in the current build system, but will be able to in the future one. (Feature rescheduled for F20.)
Why not use fstrim on the image file?
Um. Because no good reason? This is both fast and produces the smallest size. There was some problem with it months ago, and I had discarded it as an option. But clearly that was wrong!
I remembered / re-discovered the reason: it fails in virt-install with "fstrim: /: FITRIM ioctl failed: Operation not supported".
I should probably figure out *why* rather than just avoiding it, though. :)
On Tue, 21 May 2013, Bill Nottingham wrote:
Matthew Miller (mattdm@fedoraproject.org) said:
- I also commented out the "Zeroing out empty space"
postinstall stuff, because it drastically increases the image build time for not much benefit, IMHO.
One time image build cost vs. whatever benefit multipled by every time the image is used. :)
To put some numbers behind it, the compressed qcow2 image with the dd to zero empty space is 215M out of appliance-creator. Without it, it's 242M.
post install is the wrong place, agreed. Pre-build of an image is not
There are privacy implications in not blanking a VM image. In some LVM setups, one may pick up a previous image's slack space still containing live data. A cautious provider does not provide an image with anything worth trolling (trawling <?>) through
Matt,
What timing is seen running (pre install): shred -n 0 -z /path/to/image compared to that 'dd' approach quoted?
-- Russ herrold
On Tue, May 21, 2013 at 5:34 PM, Matthew Miller mattdm@fedoraproject.orgwrote:
On Wed, May 15, 2013 at 11:18:04AM -0400, Matthew Miller wrote:
- I also commented out the "Zeroing out empty space" postinstall
stuff,
because it drastically increases the image build time for not much
benefit,
IMHO.
One time image build cost vs. whatever benefit multipled by every time
the
image is used. :)
To put some numbers behind it, the compressed qcow2 image with the dd to zero empty space is 215M out of appliance-creator. Without it, it's 242M.
Two things here: First, I did not mean to imply that I thought zeroing out the disk was a completely useless operation. I merely was not willing to pay the penalty for building my test image, and I was trying to be completely honest about exactly how I built my image. Second, I think one of the reasons this operation annoys me is that I'm used to using ami-creator to create filesystem images (rather than full-disk images), and in that scenario I start with a very small filesystem (1 GB or less) and grow it after it's created. The idea of having a 10 GB disk, filling less than 3% of its capacity, and then having to write zeros to the other 97%, most of which was not actually needed for the install, seems pretty suboptimal. If there are tools to optimize it (fstrim or whatever), that's great. Otherwise, I'd be considering some hack like: make a small partition on the disk, do the install, zero out bytes on the partition, and then repartition.
I'm all for minimizing the size of whatever we make available for download. I also want people to feel like they can do their own test builds without killing their hard drive. :-)
Andy
If we use a smaller blocksize, or an incrementally shrinking one, we can
shrink it by another megabyte, at the cost of even more build time. I think that's probably across the worth-it threshold, though.
Using virt-sparsify seems to take even longer, but does get it down to 208M. That requires libguestfs, which we can't do in the current build system, but will be able to in the future one. (Feature rescheduled for F20.)
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ < mattdm@fedoraproject.org> _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
On Wed, May 22, 2013 at 2:54 AM, Andy Grimm agrimm@gmail.com wrote:
On Tue, May 21, 2013 at 5:34 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, May 15, 2013 at 11:18:04AM -0400, Matthew Miller wrote:
- I also commented out the "Zeroing out empty space" postinstall
stuff, because it drastically increases the image build time for not much benefit, IMHO.
One time image build cost vs. whatever benefit multipled by every time the image is used. :)
To put some numbers behind it, the compressed qcow2 image with the dd to zero empty space is 215M out of appliance-creator. Without it, it's 242M.
Two things here: First, I did not mean to imply that I thought zeroing out the disk was a completely useless operation. I merely was not willing to pay the penalty for building my test image, and I was trying to be completely honest about exactly how I built my image. Second, I think one of the reasons this operation annoys me is that I'm used to using ami-creator to create filesystem images (rather than full-disk images), and in that scenario I start with a very small filesystem (1 GB or less) and grow it after it's created. The idea of having a 10 GB disk, filling less than 3% of its capacity, and then having to write zeros to the other 97%, most of which was not actually needed for the install, seems pretty suboptimal. If there are tools to optimize it (fstrim or whatever), that's great. Otherwise, I'd be considering some hack like: make a small partition on the disk, do the install, zero out bytes on the partition, and then repartition.
And that's exactly what cloud-init can be used for. We're building our images with 2G disk size and let cloud-init grow them to the max size at boot which can be different for different flavors or if a user creates a bootable volume with an arbitrary size.
...Juerg
I'm all for minimizing the size of whatever we make available for download. I also want people to feel like they can do their own test builds without killing their hard drive. :-)
Andy
If we use a smaller blocksize, or an incrementally shrinking one, we can shrink it by another megabyte, at the cost of even more build time. I think that's probably across the worth-it threshold, though.
Using virt-sparsify seems to take even longer, but does get it down to 208M. That requires libguestfs, which we can't do in the current build system, but will be able to in the future one. (Feature rescheduled for F20.)
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
El Tue, 21 May 2013 20:54:25 -0400 Andy Grimm agrimm@gmail.com escribió:
On Tue, May 21, 2013 at 5:34 PM, Matthew Miller mattdm@fedoraproject.orgwrote:
On Wed, May 15, 2013 at 11:18:04AM -0400, Matthew Miller wrote:
- I also commented out the "Zeroing out empty space"
postinstall
stuff,
because it drastically increases the image build time for not much
benefit,
IMHO.
One time image build cost vs. whatever benefit multipled by every time
the
image is used. :)
To put some numbers behind it, the compressed qcow2 image with the dd to zero empty space is 215M out of appliance-creator. Without it, it's 242M.
Two things here: First, I did not mean to imply that I thought zeroing out the disk was a completely useless operation. I merely was not willing to pay the penalty for building my test image, and I was trying to be completely honest about exactly how I built my image. Second, I think one of the reasons this operation annoys me is that I'm used to using ami-creator to create filesystem images (rather than full-disk images), and in that scenario I start with a very small filesystem (1 GB or less) and grow it after it's created. The idea of having a 10 GB disk, filling less than 3% of its capacity, and then having to write zeros to the other 97%, most of which was not actually needed for the install, seems pretty suboptimal. If there are tools to optimize it (fstrim or whatever), that's great. Otherwise, I'd be considering some hack like: make a small partition on the disk, do the install, zero out bytes on the partition, and then repartition.
its actually completely wasted in the current processes. koji desparsifies the images when uploading to the hub. so we end up with 10gb files anyway
Dennis
On Thu, May 23, 2013 at 03:29:44PM -0500, Dennis Gilmore wrote:
its actually completely wasted in the current processes. koji desparsifies the images when uploading to the hub. so we end up with 10gb files anyway
No that's different. This trims junk _which isn't necessarily zeros_ from unused (but previously used) blocks. Without it, the 10gb file compresses to 242MB, but with it, to 208MB.
You can build them yourself: https://git.fedorahosted.org/cgit/cloud-kickstarts.git/
Are there any plans for OpenStack images? The Ec2 kickstart probably works but it is a little too Amazonian for an OpenStack image.
...Juerg
On Fri, May 10, 2013 at 12:36 PM, Juerg Haefliger juergh@gmail.com wrote:
On Fri, May 10, 2013 at 5:22 PM, Krishna Raman kraman@gmail.com wrote:
Can I get a hold of the test image?
I'd be interested too.
...Juerg
-kr
On May 10, 2013 8:20 AM, "Matthew Miller" mattdm@fedoraproject.org wrote:
On Thu, May 09, 2013 at 04:18:45PM -0700, Krishna Raman wrote:
Wondering if there is a Fedora 19 AMI available on EC2 US-East-1? I would like to start building and testing OpenShift Origin on there if possible.
There were some testing ones but we didn't announce an official one, which we do plan to do for the beta.
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org _______________________________________________ cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
cloud mailing list cloud@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/cloud
On Tue, May 14, 2013 at 04:44:33PM +0200, Juerg Haefliger wrote:
Are there any plans for OpenStack images? The Ec2 kickstart probably works but it is a little too Amazonian for an OpenStack image.
I'm working to get rid of the differences, and in fact just changed the EC2 kickstart to be a symlink to the generic cloud one. (This has the extlinux bootloader installed, which won't be used in EC2 but is small enough that I think doesn't hurt and is worthwhile in the name of keeping these differences down.)
The remaining big things that are EC2-ish are:
- ec2-user as default username, which I'm still open to discussion about. - a kludge on the i386 image for performance under Xen (bugzilla #708406)
cloud@lists.stg.fedoraproject.org