Tuesday, January 31, 2012

Image creation, part deux

My last blog post was a long and quite hackish procedure for running a Fedora install on a live instance in a Eucalyptus 3 cloud... and now I'm going to show you the easier way to build an image.  I spent some time kicking around ami-creator, and I only ran into a few small issues.  I've forked it on github and committed the necessary changes.  There is a sample kickstart file in the source tree.  Installation is a snap (sorry for not having it in rpm form, but that wasn't the goal of the day):
  • easy_install ez_setup
  • git clone https://github.com/eucalyptus/ami-creator
  • cd ami-creator
  • python setup.py build
  • python setup.py install
  • mkdir ~/f16-image
  • cp ks-fedora-16.cfg ~/f16-image/
  • cd ~/f16-image
  • optionally, go modify the kickstart file to point to your mirror, add the packages you want, change the disk size, etc.
  • ami-creator -c ks-fedora16.cfg -n f16test -v -e
When  the process completes, you'll have a few new files in the current directory:
  • f16test.img
  • initramfs-3.2.2-1.fc16.x86_64.img
  • initrd-plymouth.img
  • vmlinuz-3.2.2-1.fc16.x86_64
You can ignore initrd-plymouth.img.  Just go through the normal steps of bundle, upload, and register for each of the other three files, and you should have a working Fedora EMI.  It can't get much simpler than that.  Thanks to Jeremy Katz for starting the ami-creator project.  I hope that someday we'll see this rolled into the live image tools project where it belongs.

Image creation in the cloud

This post is the result of a challenge given to me by Seth Vidal, which showed up in his weekend blog post.  He was musing about whether it's possible to actually do a kickstart, or even an interactive install, in a cloud instance. I have to put some disclaimers around this post, because I am _not_ advocating this approach, and I'm going to show you a feature of Eucalyptus 3 that could void your warranty if used in anger. As my friend Michael likes to say, if you break it, you get to keep both pieces.

What we hashed out on Friday was that, in order to be able to kickstart inside an instance, you have to be able to pass boot parameters. In Eucalyptus 2, the only real way to do this was by patching the node controller with something similar to the NEuca patches. In Eucalyptus 3, we've implemented a sort of "escape hatch" called nc-hooks to allow folks to customize behaviors at instance definition and launch time. There's an example shell script in /etc/eucalyptus/nc-hooks/ which shows how you might write your own hooks.

Knowing that the nc-hooks feature existed, I had to think about exactly how to pass boot parameters and get them into libvirt.xml before instance launch. Passing them via userData was the obvious choice.  I came up with a couple of xslt files and this script to make the magic happen:



rewrite_libvirt_xml() {
  # Get only the value of the "bootparams=..." line from userData
  BP=$( xsltproc $euca_scripts/get-user-data.xsl $inst_home/instance.xml \
       | base64 -d \
       | sed -r "/bootparams=/!d; s/^.*bootparams=(.*)/\1/" || exit 1 )

  # Substitute the value of $BP into the stylesheet
  sed -e "s!@@BOOTPARAMS@@!$BP!" < $euca_scripts/insert-boot-params.xsl \
       > $inst_home/insert-boot-params.xsl || exit 2
  # Rewrite and replace libvirt.xml for this instance
  xsltproc $inst_home/insert-boot-params.xsl $inst_home/libvirt.xml \
       > $inst_home/libvirt.xml.new || exit 3
  cp $inst_home/libvirt.xml $inst_home/libvirt.xml.orig 
  mv -f $inst_home/libvirt.xml.new $inst_home/libvirt.xml 

case "$event" in
        exit 0
        exit 0
I don't have a vast amount of experience when it comes to xml processing, so forgive the horror of these stylesheets. The first one, get-usr-data.xsl, is quite simple:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
    <xsl:output encoding="UTF-8" indent="yes" method="text"/>
    <xsl:template match="/instance">
        <xsl:value-of select="/instance/userData"/>

The second is a little stranger, and was done with some help from StackOverflow:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:output encoding="UTF-8" omit-xml-declaration="yes" indent="yes" method="xml"/>
  <xsl:template match='node()|@*'>
      <xsl:apply-templates select='node()|@*'/>
  <xsl:template match="cmdline">

So with these files in place, I now need to configure an installer kernel and ramdisk.  These come from the /fedora/releases/16/Fedora/x86_64/os/images/pxeboot/ directory of your favorite Fedora mirror site.  The kernel and ramdisk registration process is the usual:

  • euca-bundle-image --kernel true -i vmlinz
  • euca-upload-bundle -b f16 -m /tmp/vmlinuz.manifest.xml
  • euca-register f16/vmlinuz.manifest.xml
  • euca-bundle-image --ramdisk true -i initrd.img
  • euca-upload-bundle -b f16 -m /tmp/initrd.img.manifest.xml
  • euca-register f16/initrd.img.manifest.xml
I don't really *need* a disk image here, but an EMI cannot be registered without one, so I fake it:
  • dd if=/dev/zero of=fake-emi.img bs=1k count=10000
  • mke2fs fake-emi.img
  • euca-bundle-image -i fake-emi.img
  • euca-upload-bundle -b f16 -m /tmp/fake-emi.img.manifest.xml
  • euca-register --kernel eki-EA183EA8 --ramdisk eri-6ED23EF2 f16/fake-emi.img.manifest.xml
Note that even if you aren't trying to do key injection, the disk image needs to have an ext2-compatible filesystem on it.

Next, I need a volume to install into:
  • euca-create-volume -s 10 -z PARTI00
Before I boot the instance, here's where I have to be honest with you readers: I make a lot of mistakes when I test things like this.  Typos, logic errors, you name it. So for debugging purposes, I uncomment this line in /etc/eucalyptus/libvirt.xsl:

<graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen=''/>

You definitely should not have this line uncommented for normal use, as it will allocate a port for vnc for every instance you launch, and without some extra configuration, it doesn't even require a password to connect. For quick debugging on a safe network, though, it's a good way to see what's going wrong during the boot process.

Now to launch my installer instance:

euca-run-instances -t m1.xlarge \
      -d "bootparams=ksdevice=link ip=dhcp vnc keymap=us lang=en_US console=ttyS0" \

This boots into an interactive install, which listens for vnc connections. Note that due to the size of the initrd, this instance needs a significant amount of RAM; I used 2GB, but 1GB would have worked. Before proceeding, I attach the volume (which I could have done via block device mapping):

euca-attach-volume -i i-447E3E89 -d sdd vol-14AE3F68

I check euca-describe-instances for the instance's IP address, connect to it with a vnc client, and proceed with the install. Once the install completes, I detach the volume and terminate the instance:

  • euca-detach-volume vol-14AE3F68
  • euca-terminate-instances i-447E3E89

Finally, I convert the volume to a snapshot and register it:
  • euca-create-snapshot vol-14AE3F68
  • euca-register -n f16-test -s snap-2CBB42D9

I boot an instance of my new EMI, and ... it fails to have a network. There were multiple problems with the networking configuration:
  1. The MAC address is hard-coded.
  2. The device name has changed from eth0 to eth1 (maybe related to #1)
  3. The NIC is configured to be controlled by NetworkManager
This is when I'm happy to have a vnc connection provided at the libvirt layer to debug the instance.  A quick setup of ifcfg-eth1 and a restart of the network gives me connectivity, and I'm up and running with a Fedora 16 instance installed entirely in the cloud.

The whole process took me about an hour or so this morning (not counting writing the xsl and shell script yesterday), and I imagine that the process would be much faster for subsequent attempts, and even faster when a kickstart is used.  Still, I'm not convinced that an approach like this has significant value over something like BoxGrinder or ami-creator.  Let the debate begin!  :-)

Thursday, January 19, 2012

Configuring Eucalyptus 3-devel

In my last entry, I explained how to checkout eucalyptus 3-devel and build it from source on Fedora 16.  This entry will explain how to follow that process with configuration and initialization of a single node cloud.

1) Configure environment variables.

export EUCALYPTUS=/opt/eucalyptus
export PATH=$PATH:$EUCALYPTUS/usr/sbin

2) Configure eucalyptus.conf -- Since this is a single node install on a network with DHCP, I am using SYSTEM mode for networking, which is the default.


3)  Set up proper file and directory permissions in the installed tree:

su -c "euca_conf --setup"

4) Initialize the database:

euca_conf --initialize

5) Create a bridge device and associate your primary NIC (this is specific to SYSTEM mode):


then restart your network

6) UPDATE: Start the CLC before getting credentials.

su -c "/opt/eucalyptus/etc/init.d/eucalyptus-cloud start"

7) Get credentials and source them:

euca_conf --get-credentials admin.zip
unzip admin.zip
source eucarc

8) Start the cloud components and register services:

euca_conf --register-walrus -H <hostname> -C walrus -P walrus
euca_conf --register-sc -H <hostname> -C SC_251 -P PARTI00
euca_conf --register-cluster -H <hostname> -C CC_251 -P PARTI00
su -c "/opt/eucalyptus/etc/init.d/eucalyptus-cc start"
euca_conf --register-nodes <hostname>
su -c "/opt/eucalyptus/etc/init.d/eucalyptus-nc start"

At this point, you should have a running cloud.  To verify the components:

euca-describe-walruses ; euca-describe-storage-controllers ; euca-describe-clusters

You should see something like:

WALRUS    walrus    walrus        ENABLED    {}
CLUSTER    PARTI00            CC_251     ENABLED    {}

And to ensure that the node controller is advertising resources:

euca-describe-availability-zones verbose

which shows:

AVAILABILITYZONE    PARTI00 arn:euca:eucalyptus:PARTI00:cluster:CC_251/
AVAILABILITYZONE    |- vm types    free / max   cpu   ram  disk
AVAILABILITYZONE    |- m1.small    0004 / 0004   1    128     2
AVAILABILITYZONE    |- c1.medium    0002 / 0002   1    256     5
AVAILABILITYZONE    |- m1.large    0001 / 0001   2    512    10
AVAILABILITYZONE    |- m1.xlarge    0000 / 0000   2   1024    20
AVAILABILITYZONE    |- c1.xlarge    0000 / 0000   4   2048    20

That's all for my second post.   Comments and corrections welcome.  See you on #eucalyptus !

Building Eucalyptus 3-devel

As some readers may already be aware, Eucalyptus has started publishing code from the Eucalyptus 3 development branch on launchpad.   The build process has a fairly sizable set of dependencies, so I'd like to give a quick example of how I built and installed this code on a Fedora 16 system.  I started with a minimal x86_64 install.

1) Add Eucalyptus' yum repository which contains dependencies for the source build: 


2) Download the GPG key for verifying packages, and add it to your rpm database:

rpm --import http://downloads.eucalyptus.com/devel/gpg-keys/9d7b073c-eucalyptus-nightly-release-key.pub

3) Install dependencies.  For simplicity, this list includes build and runtime deps for all components:

yum install axis2c rampartc axis2c-devel rampartc-devel python-boto \
euca2ools libvirt-devel openssl-devel gcc java-1.6.0-openjdk-devel ant \
curl-devel libxslt-devel apache-commons-logging xalan-j2-xsltc wsdl4j \
backport-util-concurrent httpd postgresql-server libvirt PyGreSQL make \
openssh-clients scsi-target-utils qemu-kvm axis2-codegen axis2-adb-codegen

4) Install grub 1.   UPDATE: This is no longer required in 3.1.

This package is obsoleted by grub2, so it cannot be installed by yum, but it has no conflicting files, so installing it outside of the rpm database is safe:

yum install yum-utils
yumdownloader grub
cd /; rpm2cpio /root/grub-0.97-*.rpm | cpio -id
cp /usr/share/grub/x86_64-redhat/* /boot/grub/

5) Create a user named eucalyptus on your system.

useradd -G kvm eucalyptus
passwd eucalyptus

6) Disable iptables (eucalyptus must be allowed to control iptables for dynamic routing).

systemctl disable iptables.service
systemctl stop iptables.service

7) Increase shmmax on the system: 

SHMMAX=$(( 48 * 1024 * 1024 ))
echo $SHMMAX > /proc/sys/kernel/shmmax
echo "kernel.shmmax = $SHMMAX" > /etc/sysctl.d/euca_shmmax

8) Modify  /usr/lib64/axis2c/bin/tools/wsdl2c/WSDL2C.sh -- erase the existing lines and add these:

java -classpath $(build-classpath axis2/codegen axis2/kernel axis2/adb \
 axis2/adb-codegen wsdl4j commons-logging xalan-j2 xsltc \
 backport-util-concurrent ws-commons-XmlSchema ws-commons-neethi \
 ws-commons-axiom annogen ) org.apache.axis2.wsdl.WSDL2C $*

9)   Log in as the eucalyptus user to checkout and build the code

10) Check out the code from bzr:   bzr branch lp:eucalyptus && cd eucalyptus
UPDATE: Check out the code from GitHub: git clone https://github.com/eucalyptus/eucalyptus.git

11) Configure eucalyptus.  I recommend running "./configure --help"  and reading over the options, but this configuration should work:

./configure --with-axis2c=/usr/lib64/axis2c/ \

12) Modify ./clc/modules/postgresql/conf/scripts/setup_db.groovy :

Change PG_BIN on line 97 to "/usr/bin/pg_ctl"
Change PG_INITDB on line 103 to "/usr/bin/initdb"

13) Run make

Then as root, do the following:

14) Run make install

NOTE: there's an issue here which causes some files in the source tree to be root-owned after this step, so you may want to run "find . | xargs chown eucalyptus" to fix this.  Otherwise, you may see "permission denied" errors the next time you run "make" or "make distclean".

15) Copy PolicyKit configuration for libvirt into place:

mkdir -p /var/lib/polkit-1/localauthority/10-vendor.d
cp -p tools/eucalyptus-nc-libvirt.pkla \

16) Restart libvirtd:  systemctl restart libvirtd.service

I'll write up another post detailing configuration and initialization steps.  For those of you who have used eucalyptus before, you will find that the eucalyptus.conf file is mostly unchanged from 2.0.x.  The database initialization and component registration steps differ slightly, though.   Stay tuned.