AWS Advent 2014 – Finding AWS Resources Across Regions, Services, and Accounts with skew

Our first AWS Advent post comes to us from Mitch Garnaat, the creator of theAWS python library boto and who is currently herding clouds and devops over at Scopely. He’s gonna walk us through how we can discover more about our Amazon Resources using the awesome tool he’s been building, called skew.

If you only have one account in AWS and you only use one service in one region, this article probably isn’t for you. However, if you are like me and manage resources in many accounts, across multiple regions, and in many different services, stick around.

There are a lot of great tools to help you manage your AWS resources. There is the AWS Web Console, the AWSCLI, various language SDK’s like boto, and a host of third-party tools. The biggest problem I have with most of these tools is that they limit your view of resources to a single region, a single account, and a single service at a time. For example, you have to login to the AWS Console with one set of credentials representing a single account. And once you are logged in, you have to select a single region. And then, finally, you drill into a particular service. The AWSCLI and the SDK’s follow this same basic model.

But what if you want to look at resources across regions? Across accounts? Across services? Well, that’s where skew comes in.

Skew

Skew is a Python library built on top of botocore. The main purpose of skew is to provide a flat, uniform address space for all of your AWS resources.

The name skew is a homonym for SKU (Stock Keeping Unit). SKU’s are the numbers that show up on the bar codes of just about everything you purchase and that SKU number uniquely identifies the product in the vendor’s inventory. When you make a purchase they scan the barcode containing the SKU and can instantly find the pricing data for the item.

Similary, skew uses a unique identifier for each one of your AWS resources and allows you to scanthe SKU and quickly find the details for that resource. It also provides some powerful mechanisms to find sets of resources by allowing wildcarding and regular expressions within the SKU’s.

ARN’t You Glad You Are Reading This?

So, what do we use for a unique identifier for all of our AWS resources? Well, as it turns out, AWS has already solved that problem for us. Each resource in AWS can be identified by an Amazon Resource Name or ARN. The general form for ARN’s are:

So, the ARN for an EC2 instances might look like this:

This tells us the instance is in the us-west-2 region, running in the account identified by the account number 123456789012 and the instance has an instance ID of i-12345678.

Getting Started With Skew

The easiest way to install skew is via pip.

Because skew is based on botocore, as is AWSCLI, it will use the same credentials as those tools. You need to make a small addition to your ~/.aws/config file to help skew map AWS account ID’s to the profiles in the config file. Check the README for details on that.

Let’s Find Some Stuff

Once we have skew installed and configured, we can use it to find resources based on their ARN’s. For example, using the example ARN above:

Ok, that wasn’t very exciting. How do I get at my actual resource in AWS? Well, the scan method returns an ARN object and this object supports the iterator pattern in Python. This makes sense since as we will see later this ARN can actually return a lot of objects, not just one. So if we want to get our object we can:

Iterating on an ARN returns a list of Resource objects and each of these Resource objects represents one resource in AWS. Resource objects have a number of attributes like id and they also have an attribute called data that contains all of the data about that resource. This is the same information that would be returned by the AWSCLI or an SDK.

Wildcards And Regular Expressions

Finding a single resource in AWS is okay but one of the nice things about skew is that it allows you to quickly find lots of resources in AWS. And you don’t have to worry about which region those resources are in or in which account they reside.

For example, let’s say we want to find all EC2 instances running in all regions and in all of my accounts:

In that one little line of Python code, a lot of stuff is happening. Skew will iterate through all of the regions supported by the EC2 service and, in each region, will authenticate with each of the account profiles listed in your AWS config file. It will then find all EC2 instances and finally return the complete list of those instances as Resource objects.

In addition to wildcards, you can also use regular expressions as components in the ARN. For example:

This will find all DynamoDB tables in all US regions for all accounts.

Some Useful Examples

Here are some examples of things you can do quickly and easily with skew that would be difficult in most other tools.

Find all unattached EBS volumes across all regions and accounts and tally the size of wasted space.

Audit all EC2 security groups to find CIDR rules that are not whitelisted.

Find all EC2 instances that are not tagged in any way.

Building ARN’s Interactively

The ARN provides a great way to uniquely identify AWS resources but it doesn’t exactly roll off the tongue. Skew provides some help for constructing ARN’s interactively.

First, start off with a new ARN object.

Each ARN object contains 6 components:

  • scheme – for now this will always be arn
  • provider – again, for now always aws
  • service – the Amazon Web Service
  • region – the AWS region
  • account – the ID of the AWS account
  • resource – the resource type and resource ID

All of these are available as attributes of the ARN object.

If you want to build up the ARN interactively, you can ask each of the components what choices are available.

You can also try out your regular expressions to make sure they return the results you expect.

To set the value of a particular component, use the pattern attribute.

Once you have the ARN that you want, you can enumerate it like this:

Running Queries Against Returned Data

A recent feature of skew allows you to run queries against the resource data. This feature makes use of jmespath which is a really nice JSON query engine. It was originally written in Python for use on the AWSCLI but is now available in a number of other languages. If you have ever used the --queryoption of the AWSCLI, then you have used jmespath.

If you append a jmespath query to the end of the ARN (using a | as a separator) skew will send the data for each of the returned resources through the jmespath query and store the result in thefiltered_data attribute of the resource object. The original data is still available as the dataattribute. For example:

Then each resource returned would have the instance type store in the filtered_data attribute of theResource object. This is obviously a very simple example but jmespath is very powerful and the interactive query tool available on http://jmespath.org/ allows you to try your queries out beforehand to get exactly what you want.

CloudWatch Metrics

One other feature of skew is easy access to CloudWatch metrics for AWS resources. If we refer back to the very first interative session in the post, we can show how you would access those CloudWatch metrics for the instance.

We can find the available CloudWatch metrics with the metric_names attribute and then we can retrieve the desired metric using the get_metric_data method. The README for skew contains a bit more information about accessing CloudWatch metrics.

Wrap Up

Skew is pretty new and is still changing a lot. It currently supports only a subset of available AWS resource types but more are being added all the time. If you manage a lot of AWS resources, I encourage you to give it a try. Feedback, as always, is very welcome as are pull requests!