Scaling up to and beyond 10M users

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Adrian Hornsby
Cloud Architecture Evangelist – Amazon Web Services
Scaling up to and beyond 10M
users
@adhorn

Now that’s a lot
of things to read!
This is NOT
where we
want to start!

What are we scaling?

horizontally
vertically

People
Application
Network & Data
Infrastructure
Scalabilty 101

AWS Fundamentals

AWS Region and Availability Zones
Availability Zone a
Availability Zone bAvailability Zone c
Region

Region (# of Zones)
Coming Soon!
AWS Global Infrastructure
55 Availability Zones within 18 geographic Region
+ 116 Points of Presence
(105 Edge Locations and 11
Regional Edge Caches)

Let’s start from…

The “Must” from Day 1
• High quality code
• Version controlled
• Automation when possible
• Infrastructure as code
• CI/CD pipeline
• Security at every layer
• Cost conscious
• Test everything
• DR procedure
Operational Excellence

Build Measure
Learn

Users > 1

Start simple, basic

Amazon Simple Storage Service (S3)
Amazon S3

Amazon S3
App v0.1
Simple Static Website
http://poliko.adhorn.me.s3-website-us-east-1.amazonaws.com/
Bucket name: poliko.adhorn.me

Amazon Route53
• Traffic Policies
• Latency Based Routing
• Geo DNS
• Weighted Round Robin
• DNS Failover
• Private DNS for Amazon VPC
• Health Checks and Monitoring
• Domain Registration
Highly available and scalable DNS web service.

Amazon S3
App v0.2
http://poliko.adhorn.me.s3-website-us-east-1.amazonaws.com/
Simple Static Website
Amazon
Route53 http://poliko.adhorn.me

Custom backend

App v1a
Amazon
EC2
instance
Elastic IP
User
Amazon
Route 53
EC2 backend
www.example.com
54.223.92.16

App v1b
Docker
Container
Elastic IP
User
Amazon
Route 53
Containerized backend
www.example.com
54.223.92.16

Managed
API Gateway
cache
Amazon
CloudWatch
API Gateway
Endpoints on
Amazon EC2
Any other publicly
accessible endpoint
AWS Lambda
functions
Amazon
CloudFront
API Gateway
User Amazon
Route 53
App v1c
Serverless backend

Application
Elastic IP
Database
User
Amazon
Route 53
App v2.0
Adding the database www.example.com
54.223.92.16

Self-managed Fully managed
Amazon EC2 Amazon
DynamoDB
Amazon RDS
Database options
SQL vs NoSQL

User Sign-Up, Sign-In
Access Control

User Sign-Up, Sign-In, and Access Control
Amazon Cognito
Secure and scalable user
directory
Social and enterprise
identity federation

Application
Elastic IP
Database
User
Amazon
Route 53
App v2.1
with Authentication www.example.com
54.223.92.16
Amazon
Cognito

Users > 1000

Separation of content type

Content Delivery Network
Amazon CloudFront
• Cache content at the edge
• Lower load on origin
• Dynamic and static content
• Streaming video
• Custom SSL certificates
• Low TTLs

Application
Elastic IP
Database
User
Amazon
Route 53
App v3.0
Separate static assets from dynamic content
Amazon S3
Amazon
CloudFront
*.js
*.jpeg
*.mp4
Amazon
Cognito

Availability and Redundancy

Availability in Parallel
Component Availability Downtime
X 99% (2-nines) 3 days 15 hours
Two X in parallel 99.99% (4-nines) 52 minutes
Three X in parallel 99.9999% (6-nines) 31 seconds

Availability Zone 1 Availability Zone 2 Availability Zone n
Multi-AZ
Support Instance Failure
Application

• Highly available
• Health checks
• Session stickiness
• Monitoring / logging
• Content-based routing
• Container-based apps
• WebSockets
• HTTP/2
Elastic Load Balancer

Web
Instance
RDS DB Instance
Active (Multi-AZ)
Availability Zone Availability Zone
Web
Instance
RDS DB Instance
Standby (Multi-AZ)
Load
balancer
App v4.0
Available & redundant application
User
Amazon
Route 53
Amazon
CloudFront
Amazon S3
Amazon
Cognito

App v4.0
Serverless option (be aware of limitations)
Amazon
DynamoDB
AWS LambdaAmazon API
Gateway
Browser
Amazon
CloudFront
Amazon
S3
Amazon
Cognito

Caching layer

Amazon
ElastiCache
RDS DB Instance
Active (Multi-AZ)
RDS DB Instance
Standby (Multi-AZ)
ELB
User
Amazon
Route 53
Amazon
CloudFront
Amazon S3
App v5.0
Stateless application
Web
Instance
Web
Instance

Amazon Elasticache
Redis and Memcached
• Transient session data
• Shared state
• High-frequency counters
• Queues
• Leaderboards
• Lists, sets, …
In-memory data store and cache

• Microseconds response times at
millions of reads/sec from single DAX
cluster
• Reduce provisioned read capacity for
DynamoDB tables for tables with hot
data
Amazon DynamoDB Accelerator (DAX)
Read- and Write-Through
Application
DAX
DynamoDB

Users > 100,000

Sunday Monday Tuesday Wednesday Thursday Friday Saturday
Typical weekly traffic to Amazon.com

Sunday Monday Tuesday Wednesday Thursday Friday Saturday
Typical weekly traffic to Amazon.com
Provisioned capacity

November
November traffic to Amazon.com

November

November
76%
24%

Auto-Scaling • Compute efficiency
• Node failure
• Traffic spikes
• Performance bugs

November
with auto-scaling

RDS DB Instance
Active (Multi-AZ)
RDS DB Instance
Standby (Multi-AZ)
ELB
App v6.0
Auto scaling groups
User
Amazon
Route 53
Amazon
CloudFront
Amazon S3
Web
Instances
Web
Instances ElastiCache
Auto-Scaling group
STATELESS

Users > 1,000,000

Databases

Read / Write Sharding
RDS DB Instance
Read Replica
App
Instance
App
Instance
App
Instance
RDS DB Instance
Master (Multi-AZ)
RDS DB Instance
Read Replica
RDS DB Instance
Read Replica

Database Federation
Users
DB
Products
DB
App
Instance
App
Instance
App
Instance

Database Sharding
User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
CBA
App
Instance
App
Instance
App
Instance

Transient state does not belong
in the database
BUT is perfect in the cache!

Little story of Databases at Amazon
“A deep dive on how we were using our existing databases
revealed that they were frequently not used for their
relational capabilities. About 70 percent of operations were
of the key-value kind, where only a primary key was used
and a single row would be returned. About 20 percent would
return a set of rows, but still operate on only a single table.”
Werner Vogels

Prime Day 2017 Metrics
Block Storage – Use of Amazon Elastic Block Store (EBS) grew by 40% year-over-year, with aggregate
data transfer jumping to 52 petabytes (a 50% increase) for the day and total I/O requests rising to 835
million (a 30% increase).
NoSQL Database – Amazon DynamoDB requests from Alexa, the Amazon.com sites, and the Amazon
fulfillment centers totaled 3.34 trillion, peaking at 12.9 million per second.
Stack Creation – Nearly 31,000 AWS CloudFormation stacks were created for Prime Day in order to
bring additional AWS resources on line.
API Usage – AWS CloudTrail processed over 50 billion events and tracked more than 419 billion, all in
support of Prime Day.
Configuration Tracking – AWS Config generated over 14 million Configuration items for AWS
resources.

Users > 10,000,000

Asynchronous patterns

CAP Theorem
Consistency Availability Partition Tolerance
Data is consistent.
All nodes see the same state.
Every request is non-failing. Service still responds as expected
if some nodes crash.
Distributed System
In the presence of a network partition, you must
choose between consistency and availability!

Eventual Consistency
… if no new updates are
made to a given data item,
eventually all accesses to that
item will return the last
updated value.
Availability
An eventually consistent system can
return any value before it converges!!
https://en.wikipedia.org/wiki/Eventual_consistency
Distributed System
Every request is non-failing.

Process A Process B Process A Process B
Synchronous Asynchronous
Waiting
Working
Continues
get or fetch resultGet result

Non-blocking UI
https://medium.com/@sophie_paxtonUX/stop-getting-in-my-way-non-blocking-ux-5cbbfe0f0158

Message passing for async. patterns
A
Queue
B
A
Queue
BListener
Pub-Sub

Web
Instances
Worker
Instance
Worker
Instance
Queue
API
Instance
API
Instance
API
Instance
API: {DO foo}
PUT JOB: {JobID: 0001, Task: DO foo}
API: {JobID: 0001}
GET JOB: {JobID: 0001, Task: DO foo}
Cache
Result:
{
JobID: 0001,
Result: bar
}

Worker
Instance
Worker
Instance
Queue
API
Instance
API
Instance
API
Instance
Cache
Amazon SNS
Push Notification
User

RDS DB Instance
Active (Multi-AZ)
Availability Zone
Elastic Load
Balancer
Web
Instance
Web
Instance
Amazon
Route 53User
Amazon S3
Amazon
Cloudfront
ElastiCache
Worker
Instance
Worker
Instance
App v7.0
Full Decoupling
Queue Amazon SNS

The Microservices Architecture

Event-driven patterns to the rescue

How Lambda works
S3 event
notifications
DynamoDB
Streams
Kinesis
events
Cognito
events
SNS
events
Custom
events
CloudTrail
events
LambdaDynamoDB
Kinesis S3
Any custom
Invoked in response to events
- Changes in data
- Changes in state
Redshift
SNS
Access any service,
including your own
Such as…
Lambda functions
CloudWatch
events

Event-driven using Lambda and S3
AWS Lambda:
Resize Images
Users upload photos
S3:
Source Bucket
S3:
Destination Bucket
Triggered on
PUTs

Event-driven using Lambda and DynamoDB
Amazon
DynamoDB
AWS
Lambda
Amazon
DynamoDB Streams
Actions
Item Create
Item Update
Item Delete

Monitoring & Steady State

• Monitoring, metrics, and logging
• What are customers saying?
• Try to squeeze as much performance
out of each service/component
Make it ridiculously easy to start monitoring a new metric!
1 line of code rule

Steady State
• ”normal” behavior of your system
https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero

Steady State
• ”normal” behavior of your system
• Business Metric
https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a

Increasing Resiliency

Circuit Breaker
• Wrap a protected function call in a
circuit breaker object, which
monitors for failures.
• If failures reach a certain
threshold, the circuit breaker trips.
https://martinfowler.com/bliki/CircuitBreaker.html

Service Degradation & Fallbacks

Save yourself a rollback ..

Immutable Infrastructure
No updates on live systems
• Always start from a new resource being provisioned
• Deploy the new software
• Test in different environments (dev, staging)
• Deploy to prod (inactive)
• Change references (DNS or Load Balancer)
• Keep old version around (inactive)
• Fast rollback if things go wrong

New Version
Users
Canary deployment for Immutable Infra.
Old Version
99%
Users
1%
Users
Start with ..
Dynamic Routing
(Route53)

Time to Go Global!

Multi-Region Multi-VPC Connectivity

S3 - Cross-Region Replication

Cross-Region Read Replicas for Amazon RDS
** For Aurora, MySQL,
MariaDB and PostgreSQL
engines.

Cross-Region Usage Pattern
• Regional Reads
• All critical writes traffic
to a single master

Aurora multi-master - scale out reads & writes
First MySQL compatible DB service with scale-out across multiple data centers
Zero application downtime from ANY instance failure
Zero application downtime from ANY AZ failure
Faster write performance and higher scale
Sign up for single-region multi-master preview today;
Multi-Region Multi-Master coming in 2018
Availability
Zone 1
Scale out both reads and writes
Availability
Zone 2
Availability
Zone 3
Application
Read/Write
Master 1
Shared distributed storage volume
Read/Write
Master 2
Read/Write
Master 3

Amazon DynamoDB Global Tables (GA)
First fully managed, multi-master, multi-region database
Build high performance, globally distributed applications
Low latency reads & writes to locally available tables
Disaster proof with multi-region redundancy
Easy to set up and no application rewrites required
Globally dispersed users
Replica (N. America)
Replica (Europe)
Replica (Asia)
Global App
Global Table

Amazon
DynamoDB
Gateway
Amazon
DynamoDB
Gateway
Amazon
Route53
eu-west-1
us-east-1
GlobalTables
https://globalddb.adhorn.me/

Users beyond 10,000,000

… at the Edge

Fire Drills

Testing to the next level: Failure Injection
• Start small & build confidence
• Application level
• Host failure
• Resource attacks (CPU, memory, …)
• Network attacks (dependencies, latency, …)
• Region attacks!

Chaos Engineering

Thank you!
@adhorn
https://medium.com/@adhorn

Scaling up to and beyond 10M users

Recommended

Recommended

More Related Content

More from Adrian Hornsby

More from Adrian Hornsby (20)

Recently uploaded

Recently uploaded (20)

Scaling up to and beyond 10M users