When the Angry CFO Comes Calling: AWS Cost Control

24. December 2016 2016 0

Author: Corey Quinn
Editors: Jesse Davis

Controlling costs in AWS is a deceptively complex topic — as anyone who’s ever gone over an AWS billing statement is sadly aware. Individual cost items in Amazon’s cloud environments seem so trivial– 13¢ an hour for an EC2 instance, 5¢ a month for a few files in an S3 bucket… until before you realize it, you’re potentially spending tens of thousands of dollars on your AWS infrastructure, and your CFO is turning fascinating shades of purple. It’s hard to concentrate on your work over the screaming, so let’s take a look into fixing that.

There are three tiers of cost control to consider with respect to AWS.

First Tier

The first and simplest tier is to look at your utilization. Intelligent use of Reserved Instances, ensuring that you’re sizing your instances appropriately, validating that you’re aware of what’s running in your environment– all of these can unlock significant savings at scale, and there are a number of good ways to expose this data. Cloudability, CloudDyn, CloudCheckr, and other services expose this information, as does Amazon’s own Trusted Advisor– if you’ve opted to pay for either AWS’s Business or Enterprise support tiers. Along this axis, Amazon also offers significant discounting once you’re in a position where signing an Enterprise Agreement makes sense.

Beware: here be dragons! Reserved Instances come in both 1 and 3 year variants– and the latter is almost always inappropriate. By locking in pricing for specific instances types, you’re opting out of three years of AWS price reductions– as well as generational improvements in instances. If Amazon releases an instance class that’s more appropriate for your workload eight months from your purchase of a 3 year RI, you get twenty-eight months of “sunk cost” before a wholesale migration to the new class becomes viable. As a rule of thumb, unless your accounting practices force you into a three year RI model, it’s best to pass them up; the opportunity cost doesn’t justify the (marginal) savings you get over one year reservations.

Second Tier

This is all well and good, but it only takes you so far. The second tier of cost control includes taking a deeper dive into how you’re using AWS’s services, while controlling for your business case. If you have a development environment that’s only used during the day, programmatically stopping it at night and starting it again the following morning can cut your costs almost in half– without upsetting the engineers, testers, and business units who rely on that environment.

Another example of this is intelligent use of Spot Instances or Spot Fleets. This requires a bit of a deep dive into your environment to determine a few things, including what your workload requirements are, how you’ve structured your applications to respond to instances joining or leaving your environment at uncontrolled times, and the amount of engineering effort required to get into a place where this approach will work for you. That said, if you’re able to leverage Spot fleets, it unlocks the potential for massive cost savings– north of 70% is not uncommon.

Third Tier

The third tier of cost control requires digging into the nature of how your application interacts with AWS resources. This is highly site specific, and requires an in-depth awareness of both your application and AWS work. “Aurora looks awesome for this use case!” without paying attention to your IOPS can result in a surprise bill for tens of thousands of dollars per month– a most unwelcome surprise for most companies! Understanding not only how AWS works on the fringes, but understanding what your application is doing becomes important.

Depending upon where you’re starting from, reducing your annual AWS bill by more than half is feasible. Amazon offers many opportunities to save money; your application architecture invariably offers many more. By tweaking these together, you can realize the kind of savings that both you and your CFO’s rising blood pressure can both enjoy.

About the Author

Principal at The Quinn Advisory Group, Corey Quinn has a history as an engineering manager, public speaker, and advocate for cloud strategies which speak to company culture. He specializes in helping companies control and optimize their AWS cloud footprint without disrupting the engineers using it. He lives in San Francisco with his wife, two dogs, and as of May 2017 his first child.

 

Securing Machine access in AWS

23. December 2016 2016 0

Authors: Richard Ortenberg, and Aren Sandersen

Hosting your infrastructure in AWS can provide numerous operational benefits, but can also result in weakened security if you’re not careful. AWS uses a shared responsibility model in which Amazon and its customers are jointly responsible for securing their cloud infrastructure. Even with Amazon’s protections, the number of attack vectors in a poorly secured cloud system is practically too high to count: password lists get dumped, private SSH keys get checked in to GitHub, former employees reuse old credentials, current employees fall victim to spear-phishing, and so on. The most critical first steps that an organization can take towards better security in AWS is putting its infrastructure in a VPN or behind a bastion host and improving its user host access system.

The Edge

A bastion host (or jump box) is a specific host that provides the only means of access to the rest of your hosts. A VPN, on the other hand, lets your computer into the remote network, allowing direct access to hosts. Both a VPN and bastion host have their strengths and weaknesses, but the main value they provide is funnelling all access through a single point. Using this point of entry (or “edge”) to gain access to your production systems is an important security measure. If your endpoints are not behind an edge and are directly accessible on the internet, you’ll have multiple systems to patch in case of a zero-day and each server must be individually “hardened” against common attacks. With a VPN or bastion, your main concern is only hardening and securing the edge.

If you prefer to use a bastion host, Amazon provides an example of how to set one up: https://aws.amazon.com/blogs/security/how-to-record-ssh-sessions-established-through-a-bastion-host/

 

If you’d rather run a VPN, here are just a few of the more popular options:

  • Run the open-source version of OpenVPN which is available in many Linux distributions.
  • Use a prebuilt OpenVPN Access Server (AS) in the AWS Marketplace. This requires a small license fee but set up and configuration are much easier.
  • Use the Foxpass VPN in AWS Marketplace.

Two Factor Authentication

One of the most critical security measures you can implement next is to configure two-factor authentication (2FA) on your VPN or bastion host. Two-factor authentication requires that users enter a code or click a button on a device in their possession to verify a login attempt, making unauthorized access difficult.

Many two-factor systems use a smartphone based service like Duo or Google Authenticator. Third party devices like RSA keys and Yubikeys are also quite common. Even if a user’s password or SSH keys are compromised, it is much harder for an attacker to also gain access to a user’s physical device or phone. Additionally, these physical devices are unable to be stolen remotely, decreasing an attack vector by multiple orders of magnitude.

For 2FA, bastion hosts use a PAM plugin which both Duo and Google Authenticator provide. If you’re using a VPN, most have built-in support for two-factor authentication.

User Host Access

Finally, you need to make sure that your servers are correctly secured behind the edge. A newly-instantiated EC2 server is configured with a single user (usually ‘ec2-user’ or ‘ubuntu’) and a single public SSH key. If multiple people need to access the server, however, then you need a better solution than sharing the private key amongst the team. Sharing a private key is akin to sharing a password to the most important parts of your infrastructure.

Instead, each user should generate their own SSH key pair, keeping the private half on their machine and installing the public half on servers which they need access to.

From easy to more complex here are three mechanisms to improve user access:

  • Add everyone’s public keys to the /home/ec2-user/.ssh/authorized_keys file. Now each person’s access can be revoked independently of the other users.
  • Create several role accounts (e.g. ‘rwuser’, ‘rouser’ that have read/write and read-only permissions, respectively) and install users’ public keys appropriately into each role’s authorized_keys file.
  • Create individual user accounts on each host. Now you have the ability to manage permissions separately for each user.

Best-practice is to use either infrastructure automation tools (e.g. Chef, Puppet, Ansible, Salt) or an LDAP-based system, (e.g. Foxpass), to create and manage the above-mentioned accounts, keys, and permissions.

Summary

There are many benefits to hosting your infrastructure in AWS. Don’t just depend on Amazon or other third parties to protect your infrastructure. Set up a VPN or bastion, patch your vulnerable systems as soon as possible, turn on 2FA, and implement a user access strategy that is more complex than just sharing a password or an SSH key.

About the Authors:

Richard Ortenberg is currently a software developer at Foxpass. He is a former member of the CloudFormation team at AWS.

Aren Sandersen is the founder of Foxpass. Previously he has run engineering, operations and IT teams at Pinterest, Bebo, and Oodle.

 

Getting Started with CodeDeploy

22. December 2016 2016 0

Author: Craig Bruce
Editors: Alfredo Cambera, Chris Castle

Introduction

While running a web site you need a way to deploy your application code in a repeatable, reliable and scalable fashion.

CodeDeploy is part of the AWS Developer Tools family which includes CodeCommit, CodePipeline, and AWS Command Line Interface. CodeCommit is for managed git repositories, CodePipeline is a service to help you build, test and automate while the AWS Command Line Interface is your best friend for accessing the API in an interactive fashion. They do integrate with each, more on that later.

Concepts

Let’s start with the CodeDeploy specific terminology:

  • Application – A unique identifier to tie your deployment group, revisions and deployments together.
  • Deployment Group – The instances you wish to deploy to.
  • Revision – Your application code that you wish to deploy.
  • Deployment – Deploy a specific revision to a specific deployment group.
  • CodeDeploy service – The managed service from AWS which oversees everything.
  • CodeDeploy agent – The agent you install on your instances for them to check in with the CodeDeploy service.

Getting up and running is straightforward. Install the CodeDeploy agent onto your EC2 instances and then head to the AWS Management Console to create an application and a deployment group. You associated your deployment group with an Auto Scaling group or EC2 tag(s). One of the new features is that on-premise instances are supported as well now. As this resource is outside of AWS’ view you need to register them with the CodeDeploy service before it can deploy to them. Your resources must have access to the public AWS API endpoints to communicate with the CodeDeploy service. This offers some really interesting options for hybrid deployment (deploying to both your EC2 and on-premise resources) – not many AWS services support any non-AWS resources. CodeDeploy is now aware of the instances which belong to a deployment group and whenever you request a deployment it will update them all.

Your revision is essentially a compressed file of your source code with one one extra file, appspec.yml, which the CodeDeploy agent will use to help unpack your files and optionally run any lifecycle event hooks you may have specified. Let’s say you need to tidy up some files before a deployment. For a Python web application you may want to remove those pesky *.pyc files. Define a lifecycle event hook to delete those files before you unpack your new code.

Upload your revision to S3 (or you can provide a commit ID from a GitHub repository, although not CodeCommit – more on this later), provide a deployment group and CodeDeploy is away. Great job, you web application code has now been deployed to your instances.

Managing deployments

As is becoming increasingly common, most AWS services are best when used with other AWS services, in this case CloudWatch offers some new options using CloudWatch Alarms and CloudWatch Events. CloudWatch Alarms can be used to stop deployments. Let’s say the CPU utilization on your instances is over 75%, this can trigger an alarm and CodeDeploy will stop any deployments on these instances. The deployment status will update to Stopped. This prevents deployments when there is an increased chance of a deployment problem.

Also new is adding triggers to your deployment groups which is powered by CloudWatch Events. An event could be “deployment succeeds” at which point a message is sent to a SNS topic. This topic could be subscribed to a Lambda function which sends a Success! message to your Deployment channel in Slack/HipChat. There are various events you can use deployment start, stop, failed as well as individual instance states, like starts, failed or succeeds. Be aware of noisy notifications though, you probably don’t want to know about every instance, in every deployment. Plus just like AWS you can be throttled by posting too many messages to Slack/HipChat in a short period.

Deployments do not always go smoothly and if there is a problem the quickest way to restore service is to revert to the last known good revision, typically the last one. Rollbacks have now been added and can be triggered in two ways. Firstly by rolling back if the new deployment fails. Secondly by rolling back if a CloudWatch Alarm is triggered. For example after a deployment if CPU usage is over 90% for 5 minutes, automatically roll the deployment back. In either case you want to know a rollback action was performed – handy that your deployment groups have notifications now.

Integration with CodePipeline

Currently CodeCommit is not a supported entry point for CodeDeploy you provide your revision via an object in S3 or a commit ID in a GitHub repository. You can however use CodeCommit as the source action for CodePipeline, behind the scenes it drops it in S3 for you before passing onto CodeDeploy. So you can build a pipeline in CodePipeline that uses CodeCommit and CodeDeploy actions. Now you have a pipeline you can add further actions as well such as integration with your CI/CD system.

Conclusion

CodeDeploy is a straightforward service to setup and a valuable tool in your DevOps toolbox. Recent updates make it easier to get notified about the status of your deployments, avoid deployments where alarms are triggered and enabling automatic rollback if there is an issue with a new deployment. Best of all use of CodeDeploy for EC2 instances is free, you only pay for your revisions in S3 (so not very much at all). If you were undecided about CodeDeploy try it today!

Notes

Read the excellent CodeDeploy documentation to see learn about all the fine details. Three features were highlighted in this post, learn more about them:

If you are new to CodeDeploy then follow the Getting Started guide to setup your IAM access and issue your first deployment.

About the Author

Dr. Craig Bruce is a Scientific Software Development Manager at OpenEye Scientific. He is responsible for the DevOps group working on Orion, a cloud-native platform for early-stage drug discovery which is enabling chemists to design new medicines. Orion is optimized for AWS and Craig is an AWS Certified DevOps Engineer.

About the Editors

Alfredo Cambera is a Venezuelan outdoorsman, passionate about DevOps, AWS, automation, Data Visualization, Python and open source technologies. He works as Senior Operations Engineer for a company that offers Mobile Engagement Solutions around the globe.

Chris Castle is a Delivery Manager within Accenture’s Technology Architecture practice. During his tenure, he has spent time with major financial services and media companies. He is currently involved in the creation of a compute request and deployment platform to enable migration of his client’s internal applications to AWS.