Amazon Simple Queue Service (Amazon SQS) makes it easy and inexpensive to enhance the scalability and reliability of your cloud application. In this session, we demonstrate design patterns for using Amazon SQS in conjunction with Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, Amazon Elastic MapReduce, Amazon Relational Database Service, and Amazon Redshift. Shazam will share their experience of combining Amazon SQS with Amazon DynamoDB to support a Super Bowl advertising campaign.

1. Speed and Reliability at Any Scale –
Combining SQS and DB Services
Jonathan Desrocher, Amazon Web Services
Colin Vipurs, Shazam Entertainment Ltd.
November 14, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. AWS Messaging = Amazon SQS + Amazon SNS

3. Amazon SQS Core Features
•
•
New and improved
Payload size of up to 256KB
Message batching for higher throughput and reduced costs
•
Supports long polling for reduced costs and latency
•
Cross-origin resource sharing support

4. SQS Core mechanics

5. Basic Message Lifecycle
Writer

6. Basic Message Lifecycle
Writer

7. Basic Message Lifecycle
Writer

8. Basic Message Lifecycle
Reader A
Reader B

9. Basic Message Lifecycle
Reader A
Reader B
ReceiveMessage
A
B

10. Basic Message Lifecycle
Reader A
Reader B
A
B

11. Basic Message Lifecycle
B
Reader B

12. Basic Message Lifecycle
Reader B

13. Basic Message Lifecycle
B
Reader B

14. Basic Message Lifecycle
Reader B

15. Basic Message Lifecycle
Reader B

16. That covers reliability.
Now let’s go for the scale!

17. Bulk Transactional Reads
Reader A

18. Bulk Transactional Reads
A
RAM: 10 Msgs
ReceiveMessage
A
A
A
A
Reader A
A
A
A
A
A

19. Bulk Transactional Reads
A
ReceiveMessage
A
A
A
A
Reader A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
RAM: 20 Msgs

20. Bulk Transactional Reads
A
ReceiveMessage
A
A
A
A
Reader A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
RAM: 30 Msgs

21. Bulk Transactional Reads
Reader A
DeleteMessage

22. Bulk Transactional Reads
Reader A

23. Let’s take it to real life!

24. Scalability example: market trade volume by half hour
350,000
300,000
250,000
200,000
150,000
100,000
50,000
0

25. Scalability example: market trade volume by half hour
350,000
300,000
250,000
200,000
85%
150,000
100,000
50,000
0
15%

26. Design pattern #1:
Batch processing

27. Batch Processing
• Use SQS as a scalable and resilient short-term storage
solution.
• Simply configure the appropriate retention period and
send away!
SendMessage
HTTP PUT
Elastic Beanstalk Application

28. Batch Processing
• When appropriate, launch a fleet of Amazon EC2
workers and process the messages en masse.
ReceiveMessage

29. Design pattern #2:
IAM Roles for Amazon EC2

30. Using IAM Roles for Amazon EC2
• Create an IAM role with the
appropriate permissions to
Amazon SQS.
• Launch EC2 instances with
this role.
• Done!
– Audit logs will correlate
the EC2 instance ID to the
SQS API calls.
– IAM will automatically
rotate the credentials on
our behalf.
{
"Statement": [
{
"Sid": "Stmt1384277213171",
"Action": [
"sqs:ChangeMessageVisibility",
"sqs:DeleteMessage",
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:ListQueues",
"sqs:ReceiveMessage"
],
"Effect": "Allow",
"Resource": "arn:aws:sqs:us-east1:455320512810:Sensor_Ingestion"
}
]
}

31. Using IAM Roles for Amazon EC2
• Use the AWS SDK on the
Instance
• No need to type credentials
•
•
•
Not in code
Not in a configuration file
Not via the console either
require ‘rubygems’
require ‘aws-sdk’
sqs = AWS::SQS.new()
myqueue = sqs.queues.named("Sensor_Ingestion")
myqueue.poll do |msg|
# Do something with the message
end

32. Design pattern #3:
Using SQS to durably batch writes

33. Using SQS to durably batch writes
• The application:
–
–
–
–
An AWS Elastic Beanstalk application.
Clients upload data to the application through HTTP PUTs.
Each upload is 100KB in size.
Amazon S3 will be used as the permanent data store.
S3 PUT
HTTP PUT
Elastic Beanstalk Application
S3
Bucket

34. Using SQS to durably batch writes
• The challenge:
– We have an external constraint that requires us to batch the upload into
Amazon S3.
For example:
• Amazon EMR best practices call for Amazon S3 object size of >10MB.
• Hourly Amazon Redshift batch inserts.
EMR Cluster
S3 PUT
HTTP PUT
Elastic Beanstalk Application
S3
Bucket
Redshift Database

35. Using SQS to durably batch writes
• Enter SQS:
– Persist individual client PUTs as SQS messages.
– Have an Amazon EC2 worker role that performs the following logic:
• Receive SQS message and add to an in-memory buffer.
• Once buffer is full, upload to Amazon S3.
• Upon acknowledgement from S3, delete SQS messages from queue.
SendMessage
HTTP PUT
Elastic Beanstalk Application
ReceiveMessage
EMR
Cluster
S3 PUT
S3
Bucket
Redshift
Database

36. Using SQS to durably batch writes
• Also to consider:
– Some data stores are optimized for read workloads
– Buffering the writes with Simple Queue Service will ensure both speed
and reliability of data ingestion.
SendMessage
HTTP PUT
Elastic Beanstalk Application
ReceiveMessage
BatchWriteItem
RDS Database

37. Design pattern #4:
Discarding stale messages

38. Discarding stale messages
• Controlled via the MessageRetentionPeriod property.
• Useful when there is no business value for data older
than X minutes.
– “Transactions that don’t complete within 5 minutes are abandoned, enabling
client-side failure handling”.
SendMessage
HTTP PUT
Elastic Beanstalk Application
ReceiveMessage
[Stale Messages]
EMR
Cluster
S3 PUT
S3
Bucket
Redshift
Database

39. Design pattern #5:
Simple Notification Service Fan-out

40. Simple Notification Service Fan-out
•
Atomically distribute a message to
multiple subscribers over different
transport methods
–
–
–
–
•
SQS queues
HTTP/S endpoints
SMS
Email
Also used to abstract different mobile
device push providers (MBL308)
– Apple Push Notification Service
– Google Cloud Messaging for Android
– Amazon Device Messaging
Publish

41. Simple Notification Service Fan-out
• Perform different operations on the same data
– Split different facets of the payload into different systems.
– Duplicate the data into short-term and longterm storage systems.
ReceiveMessage
S3 PUT
S3
Bucket
EMR
Cluster
S3
Bucket
Redshift
Database
SNS Publish
HTTP PUT
Elastic Beanstalk Application
ReceiveMessage
S3 PUT

42. Simple Notification Service Fan-out
• Deliver the same data to different environments
Production
ReceiveMessage
S3 PUT
S3
Bucket
DynamoDB
Table
S3
Bucket
DynamoDB
Table
SNS Publish
HTTP PUT
Test
Elastic Beanstalk Application
ReceiveMessage
S3 PUT

43. Simple Notification Service Fan-out
• Distribute the same data to a multiple external environments:
– Push data to different locations worldwide.
– Seamlessly synchronize AWS and on-premises environments.
– Pro tip: MessageID field is consistent across locations.
us-east-1
eu-west-1
SNS Publish
HTTP PUT
Elastic Beanstalk Application
ap-northeast-1
On-premises Data Center

44. Simple Notification Service Fan-out
• Each recipient can have its own preferred transport protocol:
– SQS for guaranteed delivery
– Email for human-friendly delivery
– HTTP/S for real-time push
Partner/Process A
Partner/Process B
SNS Publish
HTTP PUT
Elastic Beanstalk Application
Partner/Process C
Partner/Process D

45. Design pattern #6:
Send messages from the browser

46. Send messages from the browser
• Make direct calls to AWS services such as SQS
and DynamoDB directly from the user’s browser.
• Authentication is based on STS tokens.
• Supports S3, SQS, SNS and DynamoDB.

47. Send messages from the browser
• Back to our sample architecture:
– Browser authenticates against Elastic Beanstalk application
– Response includes location of SQS Queue and STS Token for direct
authentication.
GetToken
Elastic Beanstalk Application
SendMessage
ReceiveMessage
EMR
Cluster
S3 PUT
S3
Bucket
Redshift
Database

48. Colin Vipurs
Shazam Entertainment Ltd.

52. 375 MILLION
USERS
75 MILLION
MONTHLY ACTIVE USERS
10 MILLION
NEW USERS
PER MONTH

53. 2004
2005
2006
2007
2008
2009
2010
2011
2012
2013

54. Amazon SQS for surge protection
SQS
shield

55. c3po
single event
message
update request
queue worker
user data
batch
data req.
Facebook Realtime Updates
user data
user updater
DynamoDB

56. Facebook Realtime Updates

57. Queue Worker Anatomy
operation
read msg
versioned msg
data object
target system
pollers
unmarshallers
handlers

58. SQS for SLAs
data
ingester

59. SQS for SLAs
high priority
batch
data
ingester
std priority
queue worker
low priority

60. SQS as DynamoDB Buffer
data
API
DynamoDb

61. Time
Traffic Volume

62. Traffic Volume
large volume
event
Time

63. SQS as DynamoDB Buffer
data
API
DynamoDb
throughtput exceeded
data
data
data message
queue worker

64. SQS as DynamoDB Buffer
data
API
client fails
fast
DynamoDb
data
queue worker
client
retries

65. SQS as DynamoDB Buffer
public interface Writer <T> {
void write(T t);
}

66. SNS/SQS Datastore Segregation

67. SNS/SQS Datastore Segregation
Reporting
S3
tag data
API
API

68. SNS/SQS Datastore Segregation
Reporting
S3
tag data
API
API
Dev Hacking

69. SQS for Shazam is…
•
•
•
•
Protection from the outside world
Short term, unbounded persistence
Cost effective elastic capacity
Scalable data segregation

70. Thank you Colin!

71. Design patterns recap
1.
2.
3.
4.
5.
6.
Batch processing
IAM Roles for EC2
Using SQS to durably batch writes
Discard stale messages
Simple Notification Service Fan-out
Send messages from the browser

72. Additional messaging resources
• Application Services Booth
• re:Invent sessions:
– ARC301 Controlling the Flood: Massive Message Processing with
AWS SQS and DynamoDB
– MBL308 Engage Your Customers with Amazon SNS Mobile Push
• AWS Support and Discussion Forums
• AWS Architecture Center:
http://aws.amazon.com/architecture
• Documentation:
http://aws.amazon.com/documentation/sqs

73. Next stop:
ARC301 - Controlling the Flood!
Right here in this room.

74. Please give us your feedback on this
presentation
SVC206
As a thank you, we will select prize
winners daily for completed surveys!