AWS EC2 Configuration Management with Chef

Today’s post is a contribution from Joshua Timberman, a Technical Community Manager at Opscode, an avid RPGer and DM extraordinaire, a talented home-brewer, who is always Internet meme and Buzzword compliant.

He shares with us how Chef can help manage your EC2 instances.


In a previous post, we saw some examples about how to get started managing AWS EC2 instances with Puppet and Chef. In this post, we’ll take a deeper look into how to manage EC2 instances with Chef. It is outside the scope of this post to go into great detail about building cookbooks. If you’re looking for more information on working with cookbooks, see the following links

Prerequisites

There are a number of prequisites for performing the tasks outlined in this post, including

  • Workstation Setup
  • Authentication Credentials
  • Installing Chef

Workstation Setup

We assume that all commands and work will originate from a local workstation. For example, a company-issued laptop. We’ll take for granted that it is running a supported platform. You’ll need some authentication credentials, and configure knife.

Authentication Credentials

You’ll need the Amazon AWS credentials for your account. You’ll also need to create an SSH key pair to use for your instances. Finally, if you’re using a Chef Server, you’ll need your user key and the “validation” key

Install Chef

If your local workstation system doesn’t already have Chef installed, Opscode recommends using the “Omnibus package” installers.

Installing the knife-ec2 Plugin

Chef comes with a a plugin-based administrative command-line tool called knife. Opscode publishes the knife-ec2 plugin which extends knife with fog to interact with the EC2 API. This plugin will be used in further examples, and it can be installed as a RubyGem into the “Omnibus” Ruby environment that comes with Chef.

example

sudo /opt/chef/embedded/bin/gem install knife-ec2

If you’re using a different Ruby environment, you’ll need to use the proper gem command.

knife.rb

In order to use knife with your AWS account, it must be configured. The example below uses Opscode Hosted Chef as the Chef Server. It includes the AWS credentials as read in from shell environment variables. This is so the actual credentials aren’t stored in the config file directly.

Normally, the config file lives in ./.chef/knife.rb, where the current directory is a “Chef Repository.” See the knife.rb documentation for more information.

example

The additional commented lines can all be passed to the knife ec2 server create command through its options, see --help for full options list.

Launching Instances

Launch instances using knife-ec2’s “server create” command. This command will do the following:

  1. Create an instance in EC2 using the options supplied to the command, and in the knife.rb file.
  2. Wait for the instance to be available on the network, and then wait until SSH is available.
  3. SSH to the instance as the specified user (see command-line options), and perform a “knife bootstrap,” which is a built-in knife plugin that installs Chef and configures it for the Chef Server.
  4. Run chef-client with a specified run list, connecting to the Chef Server configured in knife.rb.

In this example, we’re going to use an Ubuntu 12.04 AMI provided by Canonical in the default region and availability zone (us-east–1, us-east–1d. We’ll use the default instance size (m1.small). We must specify the user that we’ll connect with SSH (-x ubuntu), because it is not the default (root). We also specify the AWS SSH keypair (-S jtimberman). As a simple example, we’ll set up an Apache Web Server with Opscode’s apache2 cookbook with a simple run list (-r 'recipe[apt],recipe[apache2]'=), and use the =apt cookbook to ensure the APT cache is updated. Then, we specify the security groups so the right firewall rules are opened (-G default,www).

knife ec2 server create -x ubuntu -I ami-9a873ff3 -S jtimberman -G default,www -r 'recipe[apt],recipe[apache2]'

The first thing this command does is talk to the EC2 API and provision a new instance.

The “Bootstrap” Process

What follows will be the output of the knife bootstrap process. That is, it installs Chef, and then runs chef-client with the specified run list.

The registration step is where the “validation” key is used to create a new client key for this instance. On the Chef Server:

On the EC2 instance:

The client.pem file was created by the registration. We can safely delete the validation.pem file now, it is not needed, and there’s actually a recipe for that.

The client.rb looks like this:

The chef_server_url and validation_client_name came from the knife.rb file above. The node name came from the instance ID assigned by EC2. Node names on a Chef Server must be unique, EC2 instance IDs are unique, whereas the FQDN (Chef’s default here) may be recycled from terminated instances.

The ohai directory contains hints for EC2. This is a new feature of the knife-ec2 plugin and ohai, to help better identify cloud instances, since certain environments make it difficult to auto-detect (including EC2 VPC and Openstack).

Now that the instance has finished Chef, it has a corresponding node object on the Chef Server.

In this output, the FQDN is the private internal name, but the IP is the public address. This is so when viewing node data, one can copy/paste the public IP easily.

Managing Instance Lifecycle

There are many strategies out there for managing instance lifecycle in Amazon EC2. They all use different tools and workflows. The knife-ec2 plugin includes a simple “purge” option that will remove the instance from EC2, and if the node name in Chef is the instance ID, will remove the node and API client objects from Chef, too.

Conclusion

AWS EC2 is a wonderful environment to spin up new compute quickly and easily. Chef makes it even easier than ever to configure those instances to do their job. The scope of this post was narrow, to introduce some of the concepts behind the knife-ec2 plugin and how the bootstrap process works, and there’s much more that can be learned.

Head over to the Chef Documentation to read more about how Chef works.

Find cookbooks shared by others on the Chef Community Site.

If you get stuck, the community has great folks available via the IRC channels and mailing lists.