Taming and controlling storage volumes
Author: James Andrews
Introduction
It’s easy to use scripts in AWS to generate a lot of EC2 virtual machines quickly. Along with the creation of EC2 instances, persistent block storage volumes via AWS EBS may be created and attached. These EC2 instances might be used briefly and then discarded but by default the EBS volumes are not destroyed.
Continuous Integration
Continuous Integration (CI) is now a standard technique in software development. When doing CI it’s usually a good idea to build at least some of the infrastructure used in what are called ‘pipelines’ from scratch. Fortunately, AWS is great at this and has a lot of methods that can be used.
When CI pipelines are mentioned, Docker is often in the same sentence although Docker and other containerized environments are not universally adopted. Businesses are still using software controlled infrastructure with separate virtual machines for each host. So I’ll be looking at this type of environment which uses EC2 as separate virtual machines.
A typical CI run will generate some infrastructure such as a EC2 virtual machine, load some programs onto it, run some tests and then tear the EC2 down afterwards. It’s during this teardown process that care needs to be taken on AWS to avoid leaving attached EBS volumes existing.
AWS allows the use of multiple environments – as many as you’d like to pay for. This allows different teams or projects to run their own CI as independent infrastructure. The benefit is that testing on multiple, different streams of development is possible. Unfortunately multiple environments can also increase the problem of remnant EBS volumes that are not correctly removed after use.
Building CI Infrastructure
I’ve adapted a couple of methods for making CI infrastructure.
1. Cloudformation + Troposphere
AWS CloudFormation is a great tool but one drawback is the use of JSON (or now yaml) configuration files to drive it. These are somewhat human readable but are excessively long and tend to contain “unrolled” repeated phrases if several similar EC2 instances are being made.
To mitigate this problem I have started using Troposphere, which is a python module to generate CloudFormation templates. I should point out that similar CloudFormation template making libraries are available in other languages, including Go and ruby.
It may seem like just more work to use a program to make what is essentially another program ( the template ) but by abstracting the in house rules for making an EC2 just how you need it, template development is quicker and the generator programs are easier to understand than the templates.
Here is an example troposphere program:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
from troposphere import Ref, Template from troposphere import Base64, GetAtt, Join from troposphere.route53 import RecordSetType import troposphere.ec2 as ec2 def doInstance(itype, hostname, iname, customscript=None, ami='ami-7abd0209', vsize="8"): """ itype is a instance size like "m4.large" """ """ hostname can be a Ref, such as from a parameter """ """ iname is a internal name for the instance in the CF template""" """ ami is the optional ami name.""" """ Default ami a Centos7 image in eu-west-1 """ """ vsize is optional EBS volume size in GB. Default is 8.""" """ Note not all ami support resizing to match the supplied EBS """ if (customscript is not None): script = customscript fqdn = Join('', [hostname, '.', basedomain, '.']) instance = ec2.Instance(iname) instance.UserData = Base64(Join('', [ "#!/bin/bash", "\n", "yum install -y bind-utils", "\n", script, "\n", "exit 0", "\n"])) instance.ImageId = ami instance.InstanceType = itype # automatically set up DNS instance.KeyName = keyname dns = RecordSetType( "myDNSRecord%s" % iname, HostedZoneName="%s." % basedomain, Comment="DNS name ", Name=fqdn, Type="A", TTL="900", ResourceRecords=[GetAtt(instance, "PrivateIp")]) t.add_resource(instance) t.add_resource(dns) if __name__ == '__main__': basedomain = 'awsadvent.com' keyname = 'my-alreadyexisting-key' t = Template() doInstance('t2.medium', 'www', 'frontend', customscript="""yum install -y httpd""") doInstance('t2.medium', 'm1', 'microservicesa', customscript="""yum install -y jdk-1.8.0""") doInstance('t2.medium', 'm2', 'microservicesb', customscript="""yum install -y jdk-1.8.0""") doInstance('t2.medium', 'broker', 'rabbitmq', customscript="""yum install rabbitmq""") t.add_description(""" www, 2x microservices and rabbitmq """) print(t.to_json()) |
When the program is run it prints a json template to stdout.
Once the templates are made they can either be deployed by the console or AWS cli.
2. Boto and AWS cli
There is also a python module to directly call the AWS API. This module is called boto. By using this the EC2 can be made directly when running the boto program. Again, similar libraries are available in other languages. With the troposphere generator, there is an intermediate step but boto programs directly make the AWS infrastructure items.
If you have used AWS cli, boto is very similar but allows all the inputs and outputs to be manipulated in python. Typical boto programs are shorter than AWS cli shell programs and contain less API calls. This is as it is much easier to get the required output data in python than in shell or the AWS cli.
Monitoring EBS volumes
If you are managing an environment where it is difficult to apply controls to scripts then there can be a buildup of “dead volumes” that are not attached to any live instance. Some of these might be of historical interest but mostly they are just sitting there accruing charges.
Volumes that should be kept should be tagged appropriately so that they can be charged to the correct project or deleted when their time is past. However, when scripts are written quickly to get the job done, often this is not the case.
This is exactly the situation that our team found ourself in following an often frantic migration of a legacy environment to AWS. To audit the volumes, I wrote this boto script. It finds all volumes that are not attached to an EC2 and which have no tags. This script is running as a cronjob on a management node in our infrastructure.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
import boto3 import datetime target = 'default' # target account if there are multiple accounts # load session credentials mysession = boto3.Session(profile_name=target) enp = mysession.client('ec2') enr = mysession.resource('ec2') # query all AWS EC2 EBS volumes in this account resp = enp.describe_volumes() deadlist = [] # find all volumes that are not attached and have no tags for x in resp['Volumes']: if len(x['Attachments']) == 0 and 'Tags' not in x: deadlist.append(x) # if there are some unused volumes tag them as "NOTINUSE" with a date if len(deadlist) > 0: deadtag = [{'Key': 'NOTINUSE', 'Value': '{:%Y-%m-%d %H:%M:%S}' .format(datetime.datetime.now())}] # extract just the VolumeIds into a list rlist = map(lambda x: x['VolumeId'], deadlist) print("dead vols are ", rlist) # add the tag to the dead vols enp.create_tags(Resources=rlist, Tags=deadtag) # format message msg = "%d volumes found untagged and unattached\n\n" % len(deadlist) for x in deadlist: msg = msg + repr(x) + "\n" msg = msg + "{code}\n\n" print(msg) else: print("no unused EBS volumes found: GOOD") |
3. AWS Config
A third option is to use AWS tags. This can be an important technique in a complex AWS environment being managed by a team. Tags attach metadata to almost any type of AWS object, including the EBS volumes we are looking at in this article. The tags can be used to help apply the correct policy processes to volumes to prevent wasteful use of storage.
Establishing this sort of policy is (you might have guessed) something that Amazon have thought of already. They provide a great tool unsurprisingly named the “AWS Configuration Management tool” for monitoring, tagging, and applying other types of policy rules.
The AWS Config management tool provides a framework for monitoring compliance to various rules you can set. It also will run in response to system events (like the termination of an EC2) or on a timer like cron.
It has a built in rule for finding objects that have no tags (REQUIRED_TAGS, see http://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_use-managed-rules.html ) This rule doesn’t quite do the same as my script but is a viable alternative. If you need slightly different rules to the predefined ones then AWS Config has a system for adding custom rules using Lambda programs which could alternately be used.
Stopping the problem at source
If you are writing new scripts now and want to ensure that setting delete on termination for volumes happens in your system automation scripts then the best thing to do is to add the options to make this happen in the script. EBS volumes attached to EC2 instances have a setting for “delete on termination”
The default for volume creation with CloudFormation and the API are not set to delete on termination. I was surprised to find that deleting a CloudFormation stack (which contained default settings) did not remove the EBS volumes it made.
A good way to overcome this is to set the delete on termination flag.
In AWS shell CLI, get the instanceID and add this command after a “run-instances”.
|
1 |
<span style="font-weight: 400;">aws ec2 modify-instance-attribute --instance-id $instanceID --block-device-mappings "[{\"DeviceName\":\"/dev/sda1\",\"Ebs\":{\"DeleteOnTermination\":true}}]"</span> |
In CloudFormation the Type”: “AWS::EC2::Instance” sections make volumes from the AMI automatically. To set the delete on termination add a block like this in the Properties section.
|
1 2 3 4 5 6 7 |
"BlockDeviceMappings": [ {"DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination" } } ] |
To do the same thing in troposphere make a ec2.Instance object and use the BlockDeviceMappings attribute.
|
1 2 3 |
instance.BlockDeviceMappings = [BlockDeviceMapping( DeviceName="/dev/sda1", Ebs=EBSBlockDevice(DeleteOnTermination=True))] |
This attribute could be added to the example troposphere program in the “doInstance” method at line 22 of the file.
Summary
The problem of uncontrolled EBS volume use can be brought under control by the use of tagging and using a script or AWS Config to scan for unused volumes.
Unattached, unused EBS volumes can be prevented in a software controlled infrastructure by setting the “DeleteOnTermination” flag – this does not happen by default in CloudFormation.
About the Author
James Andrews has been working for 25 years as a Systems Administrator. On the weekend he rides his bike around Wales.