Exploring Concurrency in Python & AWS
Exploring Concurrency in Python & AWS
From Threads to Lambdas (and lambdas with threads)
Author: Mohit Chawla
Editors: Jesse Davis, Neil Millard
The scope of the current article is to demonstrate multiple approaches to solve a seemingly simple problem of intra-S3 file transfers – using pure Python and a hybrid approach of Python and cloud based constructs, specifically AWS Lambda, with a comparison of the two concurrency approaches.
Problem Background
The problem was to transfer 250 objects daily, each of size 600-800 MB, from one S3 bucket to another. In addition, an initial bulk backup of 1500 objects (6 months of data) had to be taken, totaling 1 TB.
Attempt 1
The easiest way to do this appears to loop over all the objects and transfer them one by one:
|
1 2 3 4 5 6 |
for obj in src_objects: dest_obj.copy_from( CopySource={ 'Bucket': obj.bucket_name, 'Key': obj.key }) |
This had a runtime of 1 hour 45 minutes. Oops.
Attempt 2
Lets use some threads !
Python offers multiple concurrency methods:
- asyncio, based on event loops and asynchronous I/O.
- concurrent.futures, which provides high level abstractions like ThreadPoolExecutor and ProcessPoolExecutor.
- threading, which provides low level abstractions to build your own solution using threads, semaphores and locks.
- multiprocessing, which is similar to threading, but for processes.
I used the concurrent.futures module, specifically the ThreadPoolExecutor, which seems to be a good fit for I/O tasks.
Note about the GIL:
Python implements a GIL (Global Interpreter Lock) which limits only a single thread to run at a time, inside a single Python interpreter. This is not a limitation for an I/O intensive task, such as the one being discussed in this article. For more details about how it works, see http://www.dabeaz.com/GIL/.
Here’s the code when using the ThreadPoolExecutor:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
import boto3 from concurrent.futures import ThreadPoolExecutor def copy(obj, bucket, key): result = obj.copy_from(CopySource={'Bucket': bucket, 'Key': key}) return (result['ResponseMetadata']['HTTPStatusCode']) with futures.ThreadPoolExecutor(max_workers=100) as executor: todo = [] for src_object in src_objects: dest_obj = s3.Object(dest_bucket,dest_key) future = executor.submit(copy, dest_obj, src_object.bucket, src_object.key) todo.append(future) results = [] for future in futures.as_completed(todo): res = future.result() results.append(res) print(len(results)) |
This code took 1 minute 40 seconds to execute, woo !
Concurrency with Lambda
I was happy with this implementation, until, at an AWS meetup, there was a discussion about using AWS Lambda and SNS for the same thing, and I thought of trying that out.
AWS Lambda is a compute service that lets you run code without provisioning or managing servers. It can be combined with AWS SNS, which is a message push notification service which can deliver and fan-out messages to several services, including E-Mail, HTTP and Lambda, which as allows the decoupling of components.
To use Lambda and SNS for this problem, a simple pipeline was devised: One Lambda function publishes object names as messages to SNS and another Lambda function is subscribed to SNS for copying the objects.
|
1 2 3 4 5 |
__ __ __ __ __ __ __ __ __ __ __ __ | | | | | Lambda |-------> SNS ------>| Lambda | | | | | |__ __ __ __ __ __| |__ __ __ __ __ _| |
The following piece of code publishes names of objects to copy to an SNS topic. Note the use of threads to make this faster.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
import boto3 from concurrent.futures import ThreadPoolExecutor def publish_sns(client, dest_key): client.publish( TopicArn='arn:aws:sns:us-east-1:12345678:s3copy', Message=dest_key, Subject=dest_key ) return response def lambda_handler(*args): for obj in src_objects: with futures.ThreadPoolExecutor(max_workers=10) as executor: future = executor.submit(publish_sns, client, obj.key) |
Yep, that’s all the code.
Now, you maybe asking yourself, how is the copy operation actually concurrent ?
The unit of concurrency in AWS Lambda is actually the function invocation. For each published message, the Lambda function is invoked, which means for multiple messages published in parallel, an equivalent number of invocations will be made for the Lambda function. According to AWS, that number for stream based sources is given by:
|
1 |
concurrency = events per second * function duration |
By default, this is limited to 100 concurrent executions, but can be raised on request.
The execution time for the above code was 2 minutes 40 seconds. This is higher than the pure Python approach, partly because the invocations were throttled by AWS.
I hope you enjoyed reading this article, and if you are an AWS or Python user, hopefully this example will be useful for your own projects.
Note – I gave this as a talk at PyUnconf ’16 in Hamburg, you can see the slides at https://speakerdeck.com/alcy/exploring-concurrency-in-python-and-aws.
About the Author:
Mohit Chawla is a systems engineer, living in Hamburg. He has contributed to open source projects over the last seven years, and has a few projects of his own. Apart from systems engineering, he has a strong interest in data visualization.