AWS Advent 2014: Repeatable Infrastructure with CloudFormation and YAML
Ted Timmons is a long-term devops nerd and works for Stanson Health, a healthcare startup with a fully remote engineering team.
One key goal of a successful devops process – and successful usage of AWS – is to create automated, repeatable processes. It may be acceptable to spin up EC2 instances by hand in the early stage of a project, but it’s important to convert this from a manual experiment to a fully described system before the project reaches production.
There are several great tools to describe the configuration of a single instance- Ansible, Chef, Puppet, Salt- but these tools aren’t well-suited for describing the configuration of an entire system. This is where Amazon’s CloudFormation comes in.
CloudFormation was launched in 2011. It’s fairly daunting to get started with, errors in CloudFormation templates are typically not caught until late in the process, and since it is fed by JSON files it’s easy to make mistakes. Proper JSON is unwieldy (stray commas, unmatched closing blocks), but it’s fairly easy to write YAML and convert it to JSON.
EC2-VPC launch template
Let’s start with a simple CloudFormation template to create an EC2 instance. In this example many things are hardcoded, like the instance type and AMI. This cuts down on the complexity of the example. Still, it’s a nontrivial example that creates a VPC and other resources. The only prerequisite for this example is to create a keypair in the US-West-2 region called “advent2014”.
|
1 2 |
#include (<a href="https://github.com/tedder/aws-advent-2014-yml-cloudformation/blob/master/simple-ec2.json">https://github.com/tedder/aws-advent-2014-yml-cloudformation/blob/master/simple-ec2.json</a>) |
As you look at this template, notice both the quirks of CloudFormation (especially “Ref” and “Fn::GetAtt”) and the quirks of JSON. Even with some indentation the brackets are complex, and correct comma placement is difficult while editing a template.
JSON to YAML
Next, let’s convert this JSON example to YAML. There’s a quick converter in this article’s repository, with python and pip installed, the only other dependency should be to install PyYAML with pip.
|
1 2 |
./json-to-yaml.py < simple-ec2.json > simple-ec2.yml |
Since JSON doesn’t maintain position of hashes/dicts, the output order may vary. Here’s what it looks like immediately after conversion:
|
1 2 |
#include (<a href="https://github.com/tedder/aws-advent-2014-yml-cloudformation/blob/master/simple-ec2.yml">https://github.com/tedder/aws-advent-2014-yml-cloudformation/blob/master/simple-ec2.yml</a>) |
Only a small amount of reformatting is needed to make this file pleasant: I removed unnecessary quotes, combined some lines, and moved the ‘Type’ line to the top of each resource.
|
1 2 |
#include (<a href="https://github.com/tedder/aws-advent-2014-yml-cloudformation/blob/master/simple-ec2-formatted.yml">https://github.com/tedder/aws-advent-2014-yml-cloudformation/blob/master/simple-ec2-formatted.yml</a>) |
YAML to JSON to CloudFormation
It’s fairly easy to see the advantages of YAML in this case- it has a massive reduction in brackets and quotes and no need for commas. However, we need to convert this back to JSON for CloudFormation to use. Again, the converter is in this article’s repository.
|
1 2 3 |
./yaml-to-json.py < simple-ec2-formatted.yml > simple-ec2-autogenerated.json aws --region us-west-2 cloudformation create-stack --template-body file://simple-ec2-autogenerated.json --stack-name advent |
That’s it!
Ansible assembly
If you would like to use Ansible to prepare and publish to CloudFormation, my company shared an Ansible module to compile YAML into a single JSON template. The shared version of the script is entirely undocumented, but it compiles a full directory structure of YAML template snippets into a template. This significantly increases readability. Just placecloudformation_assemble in your library/ folder and use it like any other module.
If there’s interest, I’ll help to document and polish this module so it can be submitted to Ansible. Just fork and send a pull request.