Tuesday, January 31, 2012

Image creation in the cloud

This post is the result of a challenge given to me by Seth Vidal, which showed up in his weekend blog post.  He was musing about whether it's possible to actually do a kickstart, or even an interactive install, in a cloud instance. I have to put some disclaimers around this post, because I am _not_ advocating this approach, and I'm going to show you a feature of Eucalyptus 3 that could void your warranty if used in anger. As my friend Michael likes to say, if you break it, you get to keep both pieces.

What we hashed out on Friday was that, in order to be able to kickstart inside an instance, you have to be able to pass boot parameters. In Eucalyptus 2, the only real way to do this was by patching the node controller with something similar to the NEuca patches. In Eucalyptus 3, we've implemented a sort of "escape hatch" called nc-hooks to allow folks to customize behaviors at instance definition and launch time. There's an example shell script in /etc/eucalyptus/nc-hooks/ which shows how you might write your own hooks.

Knowing that the nc-hooks feature existed, I had to think about exactly how to pass boot parameters and get them into libvirt.xml before instance launch. Passing them via userData was the obvious choice.  I came up with a couple of xslt files and this script to make the magic happen:



rewrite_libvirt_xml() {
  # Get only the value of the "bootparams=..." line from userData
  BP=$( xsltproc $euca_scripts/get-user-data.xsl $inst_home/instance.xml \
       | base64 -d \
       | sed -r "/bootparams=/!d; s/^.*bootparams=(.*)/\1/" || exit 1 )

  # Substitute the value of $BP into the stylesheet
  sed -e "s!@@BOOTPARAMS@@!$BP!" < $euca_scripts/insert-boot-params.xsl \
       > $inst_home/insert-boot-params.xsl || exit 2
  # Rewrite and replace libvirt.xml for this instance
  xsltproc $inst_home/insert-boot-params.xsl $inst_home/libvirt.xml \
       > $inst_home/libvirt.xml.new || exit 3
  cp $inst_home/libvirt.xml $inst_home/libvirt.xml.orig 
  mv -f $inst_home/libvirt.xml.new $inst_home/libvirt.xml 

case "$event" in
        exit 0
        exit 0
I don't have a vast amount of experience when it comes to xml processing, so forgive the horror of these stylesheets. The first one, get-usr-data.xsl, is quite simple:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output encoding="UTF-8" indent="yes" method="text"/>
    <xsl:template match="/instance">
        <xsl:value-of select="/instance/userData"/>

The second is a little stranger, and was done with some help from StackOverflow:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output encoding="UTF-8" omit-xml-declaration="yes" indent="yes" method="xml"/>
  <xsl:template match='node()|@*'>
      <xsl:apply-templates select='node()|@*'/>
  <xsl:template match="cmdline">

So with these files in place, I now need to configure an installer kernel and ramdisk.  These come from the /fedora/releases/16/Fedora/x86_64/os/images/pxeboot/ directory of your favorite Fedora mirror site.  The kernel and ramdisk registration process is the usual:

  • euca-bundle-image --kernel true -i vmlinz
  • euca-upload-bundle -b f16 -m /tmp/vmlinuz.manifest.xml
  • euca-register f16/vmlinuz.manifest.xml
  • euca-bundle-image --ramdisk true -i initrd.img
  • euca-upload-bundle -b f16 -m /tmp/initrd.img.manifest.xml
  • euca-register f16/initrd.img.manifest.xml
I don't really *need* a disk image here, but an EMI cannot be registered without one, so I fake it:
  • dd if=/dev/zero of=fake-emi.img bs=1k count=10000
  • mke2fs fake-emi.img
  • euca-bundle-image -i fake-emi.img
  • euca-upload-bundle -b f16 -m /tmp/fake-emi.img.manifest.xml
  • euca-register --kernel eki-EA183EA8 --ramdisk eri-6ED23EF2 f16/fake-emi.img.manifest.xml
Note that even if you aren't trying to do key injection, the disk image needs to have an ext2-compatible filesystem on it.

Next, I need a volume to install into:
  • euca-create-volume -s 10 -z PARTI00
Before I boot the instance, here's where I have to be honest with you readers: I make a lot of mistakes when I test things like this.  Typos, logic errors, you name it. So for debugging purposes, I uncomment this line in /etc/eucalyptus/libvirt.xsl:

<graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen=''/>

You definitely should not have this line uncommented for normal use, as it will allocate a port for vnc for every instance you launch, and without some extra configuration, it doesn't even require a password to connect. For quick debugging on a safe network, though, it's a good way to see what's going wrong during the boot process.

Now to launch my installer instance:

euca-run-instances -t m1.xlarge \
      -d "bootparams=ksdevice=link ip=dhcp vnc keymap=us lang=en_US console=ttyS0" \

This boots into an interactive install, which listens for vnc connections. Note that due to the size of the initrd, this instance needs a significant amount of RAM; I used 2GB, but 1GB would have worked. Before proceeding, I attach the volume (which I could have done via block device mapping):

euca-attach-volume -i i-447E3E89 -d sdd vol-14AE3F68

I check euca-describe-instances for the instance's IP address, connect to it with a vnc client, and proceed with the install. Once the install completes, I detach the volume and terminate the instance:

  • euca-detach-volume vol-14AE3F68
  • euca-terminate-instances i-447E3E89

Finally, I convert the volume to a snapshot and register it:
  • euca-create-snapshot vol-14AE3F68
  • euca-register -n f16-test -s snap-2CBB42D9

I boot an instance of my new EMI, and ... it fails to have a network. There were multiple problems with the networking configuration:
  1. The MAC address is hard-coded.
  2. The device name has changed from eth0 to eth1 (maybe related to #1)
  3. The NIC is configured to be controlled by NetworkManager
This is when I'm happy to have a vnc connection provided at the libvirt layer to debug the instance.  A quick setup of ifcfg-eth1 and a restart of the network gives me connectivity, and I'm up and running with a Fedora 16 instance installed entirely in the cloud.

The whole process took me about an hour or so this morning (not counting writing the xsl and shell script yesterday), and I imagine that the process would be much faster for subsequent attempts, and even faster when a kickstart is used.  Still, I'm not convinced that an approach like this has significant value over something like BoxGrinder or ami-creator.  Let the debate begin!  :-)


  1. This is actually better than I kind of feared but if I had to bet, I would have guessed this is about where things would have been.

    That said, I suspect that what Seth *really* wants (I'll put words into his mouth, I've done it for years!) is to be able to launch an instance, have that instance go into the installer to be installed, reboot and then be the instance in question. Which is subtly but importantly different from creating an image via the installer that you then reuse.

    There shouldn't be a huge amount of work needed to make it work. And once you did then a lot of stuff starts to open up as interesting. I think it's pretty important to getting the cloud out of the world of prebaked images and instead to a far more dynamic place that can support being kickstarted dynamically, see the use of tools like crowbar and a lot more.

  2. Jeremy,
    I'm not against the idea of installer->image - start a new instance with that image - but you're correct that I don't see a compelling reason to not be able to do

    installer->run-able system

    just like I would do with any normal piece of hw or any virt-install I might run.

    In short, I do not see the reason why we are jumping through special hoops to create images at all. I can see why images are a nice convenience to an admin who wants to make 20 instances of a certain kind of system but images should only be an optimization - they should not be the primary installation/deployment mechanism. Or even if they are the primary mechanism they should not be the origin.