On 06/27/2013 12:53 AM, Collins, Robert (HPCS) wrote:

Hi, I'm the project lead for TripleO (OpenStack on OpenStack -

Hi Robert, yes, I've just noticed this project a week or so ago.

While I have your attention, probably too late now, I wonder was Oz ever considered for disk-image-building? (I've read the README of diskimage-builder briefly)

https://github.com/clalancette/oz

http://docs.openstack.org/trunk/openstack-compute/admin/content/tool-support...

Oz has been in development (& has an amicable upstream) for a couple of years, building JEOS (just enough OS) targeting multiple operating systems including Windows. It's not too RPM specific either. There's Debian packaging notes:

https://github.com/clalancette/oz/tree/master/debian

All apologies if this discussion happened elsewhere, please point me to it.

a disk image based approach to deploying and maintaining OpenStack clouds),

and we have a tool 'diskimage-builder' which consumes upstream vendor cloud images and customises them.

It would be a non-trivial nuisance if we had to expose per-cloud flavours to our users, so please consider very carefully if you go down this route: one of the major strengths of Linux is the ability to drop a filesystem image onto some hardware and have it Just Work - it's better for our users if userspace and boot tooling keep that capability, because then they don't need to respin images if they are cloudbursting across multiple providers. (Or other similar use cases).

Understood. Thanks for bringing it up.

---

From: cloud-bounces@lists.fedoraproject.org [cloud-bounces@lists.fedoraproject.org] on behalf of Kashyap Chamarthy [kchamart@redhat.com] Sent: Thursday, 27 June 2013 07:04 To: Fedora Cloud SIG Subject: Should Fedora revisit the idea having "one " image to be used across the cloud providers?

(Re-directing the discussion from #fedora-cloud on Freenode..)

Heya,

(Without trying to make it a large 'theoretical' discussion..)

So, should we (Fedora) reconsider the "noble goal" (as Matthew Miller put it) of having one cloud image serve all cloud providers? Or separate images (sure, there's more work involved) that are specific

Because, lately we've seen several problems where cloud-init has been breaking things in subtle ways, like output of boot messages on console.log for OpenStack/Amazon/Eucalyptus has been broken in different

Garret Holmstrom and Matthew Miller (both who's been in the trenches of these issues can describe much better than I can at this midnight hour). They also discussed a couple of ideas on IRC like

• configuring journald to forward to both syslog and the console

e.g. (please read the issue described above)

https://bugzilla.redhat.com/show_bug.cgi?id=977952 -- RFE: disable all direct writes to the console

And, comparing Fedora images with Cirros -- they output a neat array of bunch of commands (I think I noted this previously on this list) that's helpful for debugging. Refer this previous thread

https://lists.fedoraproject.org/pipermail/cloud/2013-April/002371.html

but haven't reached a proper conclusion.

Any further thoughts from other folks here on how to resolve this would be nice to hear.

On 06/27/2013 01:01 PM, Collins, Robert (HPCS) wrote:

Hey (and sorry about the top posting, Outlook - eesh.

Np. Unsolicited friendly suggestion - try thunderbird :)

https://bugs.launchpad.net/diskimage-builder/+bug/1182722 - Yes, we looked at Oz, and its a great tool for running installers, but that isn't what diskimage-builder does: we customise images.

Ok. (Now, I recall the proof-of-concept thread from openstack-dev, and the ensuing discussions).

So a typical debootstrap might take 10 to 15 minutes : we build an equally vanilla image in 1-2 minutes. Oz has all the complexity of dealing with Windows and other esoteric operating systems

Noted. 'debootstrap' reminds me of 'supermin' (similar for Fedora, written by Rich. Jones)

http://libguestfs.org/supermin.8.html

"Supermin is a tool for building supermin appliances. These are tiny appliances (similar to virtual machines), usually around 100KB in size, which get fully instantiated on-the-fly in a fraction of a second when you need to boot one of them."

I did a quick test previously:

http://kashyapc.fedorapeople.org/virt/openstack/supermin-fedora-appliance-bu...

From a quick 'grep' through d-i-b sources, I don't see mention of 'supermin'. Maybe I

should read more.

we do Linux and only things that can run in a chroot : so a much simpler and lighter weight tooling. Easy to debug (just drop into a shell), easy to customise (write some shell in a well known environment).

Ok, I'm yet to try disk-image-biulder to understand how it creates things. I'm just giving diskimage-builder a spin on my F19 machine.

(I personally am not really keen on windows either :) )

http://lists.openstack.org/pipermail/openstack-dev/2013-April/007185.html for instance is one thread about this on openstack-dev.

Ah, right, I followed that thread but the content of your response slipped out my head.

Right, some merits/demerits of using installer-based (anaconda/kickstart) / upstream-base iamge based.

Thanks for your response.

On Wed, Jun 26, 2013 at 07:23:44PM +0000, Collins, Robert (HPCS) wrote:

Hi, I'm the project lead for TripleO (OpenStack on OpenStack - a disk image based approach to deploying and maintaining OpenStack clouds), and we have a tool 'diskimage-builder' which consumes upstream vendor cloud images and customises them.

Why don't you use libguestfs for this?

I see you have a list of "safe" disk images, but still .. using losetup and mount to mount disk images that you download from the net on your host kernel is very dangerous. A rogue filesystem which exploits a fsdriver bug bypasses all permissions checks, process boundaries, SELinux, and the usual things that we use to keep VMs safe.

Plus libguestfs has a much cleaner API for modifying disk images, has a Python API[1] and it's already being used elsewhere in OpenStack.

Rich.

[1] http://libguestfs.org/guestfs-python.3.html

On Fri, Jun 28, 2013 at 08:37:17AM +0000, Collins, Robert (HPCS) wrote:

Then, we build a fresh filesystem, so the only thing we do with the filesystem bits we receive is copy data out of them. You are right that there is a narrow attack vector there, [but see above]; we could use guestfs's fuse support to mount and copy out the data that way.

As a note: Don't use FUSE for this (or anything if possible). libguestfs has an API for fetching a tarball from a disk image, which is far more efficient. From Python:

---------------------------------------------------------------------- #!/usr/bin/python

import sys import guestfs

assert (len (sys.argv) >= 2) disk = sys.argv[1]

g = guestfs.GuestFS (python_return_dict=True) #g.set_trace (1)

for disk in sys.argv[1:]: g.add_drive_opts (disk, readonly=1) g.launch ()

roots = g.inspect_os () if len (roots) != 1: raise (Error ("inspect_vm: no or multiple operating systems found"))

root = roots[0]

# Mount up the disks, like guestfish -i. mps = g.inspect_get_mountpoints (root) def compare (a, b): return len(a) - len(b) for device in sorted (mps.keys(), compare): try: g.mount_ro (mps[device], device) except RuntimeError as msg: print "%s (ignored)" % msg

# Export whole filesystem. g.tgz_out ("/", "/tmp/filesystem.tar.gz") ----------------------------------------------------------------------

$ ./disk2tar.py /tmp/winxp.img $ ls -lh filesystem.tar.gz -rw-rw-r--. 1 rjones rjones 2.1G Jun 28 09:49 filesystem.tar.gz

Apart from the obviously much cleaner API, libguestfs doesn't require root permissions, is more secure even for your use case, has a bunch of mature tools for "sysprepping" images, and can create disk images from tarballs.

Rich.

Just some general notes...

On 06/28/2013 03:03 PM, Dennis Gilmore wrote:

On Fri, 28 Jun 2013 08:37:17 +0000 "Collins, Robert (HPCS)" rbtcollins@hp.com wrote:

...

As far as using guestfs later in the build process - I don't see any need for that; we start with a tarball, we chroot into that, we make a filesystem matching the size of the content and rsync the contents over. The reason we start a new filesystem is that its faster: we started initially by cloning the original disk image and modifying, but it turns out that vendor images have wildly varying filesytem sizes and definitions, and we needed to provide images with well understood and documentable characteristics. Resizing ext* filesystems up and down isn't the fastest process, particularly on LTS style releases like Ubuntu LTS and RHEL.

resizing down may also impact on data layout, thus impacting future performance of the image.

so a issue with that, is that it breaks things, tar doesnt preserve filesystem capabilities so on a fedora system and presumably furture rhel things will be broken. filesystem capabilities have been taking over from setuid/setgid in packages. i pknow one thing that will be broken is that ping wont work for a user. the best way to great a image is a automated install using kickstart or whatever the vendors automated install method is

rsync -X does preserve capabilities I think.

cp -a (as root) preserves capabilities.

To get a list of capabilities on rpm based distros you could rpm --qf "[%{FILECAPS} %{FILENAMES}\n]" -qa | grep '^=' #Output at ¹ Such a list could be used later to reinstate capabilities.

cheers, Pádraig.

¹ /usr/sbin/suexec /usr/bin/gnome-keyring-daemon /usr/sbin/dumpcap /bin/ping /bin/ping6 /usr/sbin/seunshare /usr/sbin/mtr /usr/libexec/pt_chown

Heya Robyn,

On 06/28/2013 03:20 PM, Robyn Bergeron wrote:

----- Original Message -----

...

From: "Kashyap Chamarthy" kchamart@redhat.com To: "Fedora Cloud SIG" cloud@lists.fedoraproject.org Sent: Wednesday, June 26, 2013 12:04:20 PM Subject: Should Fedora revisit the idea having "one " image to be used across the cloud providers?

(Re-directing the discussion from #fedora-cloud on Freenode..)

Heya,

(Without trying to make it a large 'theoretical' discussion..)

So, should we (Fedora) reconsider the "noble goal" (as Matthew Miller put it) of having one cloud image serve all cloud providers? Or separate images (sure, there's more work involved) that are specific

As a few others have stated - I think it would be best to just have one image (to rule them all, muhahah); if nothing else, it helps to eliminate possibilities around what and why it's broken and so forth.

Yes, totally understand the nobility of one image. This thread was to get ideas from other folks to get better testing.

I also can totally see a situation where some flavors of images (for lack of a better term) eventually get so buried in little changes here and there that they just become technical debt that we will struggle to pay off if we do want to later pursue the Noble Goal.

Agreed.

I also agree with the multi-vendor, cloudbursting scenario that Robert mentioned later in this thread, and how that just makes management of images more difficult; while I would guess that a user sophisticated enough to be in the cloud is possibly creating images of their own, I still think they'd want to ensure that their starting point is consistent so that any automation they have doesn't need to be riddled with if/then scenarios.

Agreed.

...

Because, lately we've seen several problems where cloud-init has been breaking things in subtle ways, like output of boot messages on console.log for OpenStack/Amazon/Eucalyptus has been broken in different

I wonder if this is something we should be testing for?

Yes, I'm testing it for OpenStack. I presume other folks who care about Amazon/Euca does too :)

Probably an 'images' specific Fedora test day. I'm willing to volunteer.