Thursday, January 3, 2013

Using aws-cli with Eucalyptus

Just before the holidays, Amazon released awscli, a new command-line interface for managing AWS resources.  The code is based on botocore, the core python library for the next major version of boto.  I took awscli for a spin to see if it worked with the Eucalyptus Community Cloud, and as is often the case, the answer was ... almost.

First, it's useful to understand the fundamental problems that awscli was trying to address.  The most obvious is profiles.  Cloud users deal with multiple regions, accounts, users, etc., and keeping separate configurations for each one is a hassle.  awscli uses a section-based config file format which allows for multiple profiles, each of which can reference it's own region, access keys, etc.

Another problem that this new code solves is the centralization of region and service data into JSON files which are easy to read, write, and parse.  See _regions.json and _services.json in botocore for examples.

What I found was that rather than trying to alter the existing data files, what I really wanted was a eucalyptus "provider" with its own JSON files.  I'll spare you all my trial-and-error, and simply explain what worked:

  1. git clone
  2. git clone (note that this is my fork -- upstream is )
  3. Install botocore and aws-cli however you prefer ( I use "python install --user" in each directory)
  4. create a provider data directory, and a "euca" directory inside it.  I'll use /var/tmp/providers as the top directory.
  5. create _regions.json and _services.json under the "euca" directory (the linked examples here should work for ECC verbatim)
  6. symlink to botocore/data/aws/ec2.json and botocore/data/aws/iam.json in the euca provider directory
  7. Create your ~/.awsconfig file (or whatever you'd like to call it):
  8. export AWS_CONFIG_FILE=$HOME/.awsconfig
  9. export AWS_DATA_PATH=/var/tmp/providers
  10. try some commands, such as:
    aws ec2 create-volume --size 1 --availability-zone partner01
    aws ec2 describe-volumes
    aws ec2 describe-images
It may take a couple of iterations for the patch I've proposed to be accepted upstream, but in the meantime, I hope this is useful information.  As I've mentioned in the pull request, the solution is not ideal, as it requires that your default profile in a config file reference the euca provider, but I went for the least invasive fix first.  Note that even with this version, you can use profiles to group all of your eucalyptus cloud credentials into a single config file, and then have a second file for AWS profiles.  Switching back and forth is just a matter of setting AWS_CONFIG_FILE.

1 comment:

  1. Informative article on aws-cli. Thanks for providing the solution although it is not ideal. Keep up the good work and share updates. Nice post!