Author: Tal Perry
Editors: Jyrki Puttonen, Bill Weiss
What is Lambda
Side projects are my way of learning new technology. One that I’ve been anxious to try is AWS Lambda. In this article, I will focus on the things that make Lambda a great service in my opinion.
For the uninitiated, Lambda is a service that allows you to essentially upload a function and AWS will make sure the hardware is there to run it. You pay for the compute time in hundred millisecond increments instead of by the hour, and you can run as many copies of your lambda function as needed.
You can think of Lambda as a natural extension to containers. Containers (like Docker) allow you to easily deploy multiple workloads to a fleet of servers. You no longer deploy to a server, you deploy to the fleet and if there is enough room in the fleet your container runs. Lambda takes this one step further by abstracting away the management of the underlying server fleet and containerization. You just upload code, AWS containerizes it and puts it on their fleet.
Why did I choose Lambda?
My latest side project is SmartScribe, an automated transcription service. SmartScribe transcribes hours of audio in minutes, a feat which requires considerable memory and parallel processing of audio. While a fleet of containers could get the job done, I didn’t want to manage a fleet, integrate it with other services nor did I want to pay for my peak capacity when my baseline usage was far lower. Lambda abstracts away these issues, which made it a very satisfying choice.
How AWS Lambda makes me happy
It’s very cheap
I love to invest my time in side projects, I get to create and learn. Perhaps irrationally, I don’t like to put a lot of money into them from the get go. When I start building a project I want it up all the time so that I can show it around. On the other hand I know that 98% of the time, my resources will not be used.
Serverless infrastructure saves me that 98% by allowing me to pay by the millisecond instead of by the hour. 98% is a lot of savings by any account
I don’t have to think about servers
As I mentioned, I like to invest my time in side projects but I don’t like to invest it in maintaining or configuring infrastructure. A thousand little things can go wrong on your server and any one of those will bring your product to a halt. I’m more than happy to never think about another server again.
Here are a few things that have slowed me down before that Lambda has abstracted away:
- Having to reconfigure because I forgot to set the IP address of an instance to elastic and the address went away when I stopped it (to save money)
- Worrying about disk space. My processes write to the disk. Were I to use a traditional architecture I’d have to worry about multiple concurrent processes consuming the entire disk, a subtle and aggravating bug. With lambda, each function invocation is guaranteed a (small) chunk of tmp space which reduces my concern.
- Running out of memory. This is a fine point because a single lambda function can only use 1.5 G of memory.
- Applications that hold large data sets in memory might not benefit from Lambda. Applications that hold small to medium sized data sets in memory are prime candidates.
- 512MB of provisioned tmp space is a major bottleneck to writing larger files to disk.
Smart Scribe works with fairly large media files and we need to store them in memory with overhead. Even a few concurrent users can easily lead to problems with available memory – even with a swap file (and we hate configuring servers so we don’t want one). Lambda guarantees that every call to my endpoints will receive the requisite amount of memory. That’s priceless.
I use Apex to deploy my functions, which happens in one line
$ apex deploy
Apex is smart enough to only deploy the functions that have changed. And in that one line, my changes and only them reach every “server” I have. Compare that to the time it takes to do a blue green deployment or, heaven forbid, sshing into your server and pulling the latest changes.
But wait, there is more. Pardon last year’s buzzword, but AWS Lambda induces or at least encourages a microservice architecture. Since each function exists as its own unit, testing becomes much easier and more isolated which saves loads of time.
Tight integration with other AWS services
What makes microservices hard is the overhead of orchestration and communications between all of the services in your system. What makes Lambda so convenient is that it integrates with other AWS services, abstracting away that overhead.
Having AWS invoke my functions based on an event in S3 or SNS means that I don’t have to create some channel of communication between these services, nor monitor that channel. I think that this fact is what makes Lambda so convenient, the overhead you pay for a scalable, maintainable and simple code base is virtually nullified.
The punch line
One of the deep axioms of the world is “Good, Fast, Cheap : Choose two”. AWS Lambda takes a stab at challenging that axiom.
About the Author:
By day, Tal is a data science researcher at Citi’s Innovation lab Tel Aviv focusing on NLP. By night he is the founder of SmartScribe, a fully serverless automated transcription service hosted on AWS. Previously Tal was CTO of Superfly where he and his team leveraged AWS technologies and good Devops to scale the data pipeline 1000x. Check out his projects and reach out on Twitter @thetalperry
About the Editors:
Jyrki Puttonen is Chief Solutions Executive at Symbio Finland (@SymbioFinland) who tries to keep on track what happens in cloud.
Bill Weiss is a senior manager at Puppet in the SRE group. Before his move to
Portland to join Puppet he spent six years in Chicago working for Backstop
Solutions Group, prior to which he was in New Mexico working for the
Department of Energy. He still loves him some hardware, but is accepting
that AWS is pretty rad for some some things.