Do you need your applications to extend across multiple regions? Whether for disaster recovery, data sovereignty, data locality, or extremely high availability, many AWS customers choose to deploy services across regions. Join us as we explore how to design and succeed with active-active multi-region architectures. Please join us for a speaker meet-and-greet following this session at the Speaker Lounge (ARIA East, Level 1, Willow Lounge). The meet-and-greet starts 15 minutes after the session and runs for half an hour.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Architecture Patterns for Multi-
Region Active-Active Applications
A R C 2 0 9
Amy Che
Principal Solutions Delivery Manager
AWS Solutions Architecture
Darin Briskman
AWS Chief Evangelist
AWS Database, Analytics, & ML
Christopher Lane
Enterprise Architect
Chick-fil-A Corporate

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
So you want a multi-region architecture
Multi-region business continuity
Re-architecting a single region to multi-region active-active
A customer’s journey: Chick-fil-A’s path to multi-region

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What does multi-region mean to AWS?

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
multiRegionActiveActive
=
2+ Active full stack AWS Regions

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why NOT multi-region?

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why multi-region?

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why multi-region?
1. Business continuity / disaster recovery
Applications in
US-West
Applications in
US-East
Users from
San Francisco
Users from
New York
Service 1
Service 2
Service 3
Service 4
Service 1
Service 2
Service 3
Service 4

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ChinaEU-West
Users from
San Francisco
Users from
New York
Users from
London
Users from
Shanghai
Why multi-region?
1. Business continuity / disaster recovery
2. Geographically distributed customer base
US-West US-East

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why multi-region?
1. Business continuity/disaster recovery
2. Geographically distributed customer base
a. Improve latency for end-users
* Latency numbers are only examples

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why multi-region?
1. Business continuity/disaster recovery
2. Geographically distributed customer base
a. Improve latency for end-users
b. Meet legal and data regulatory compliance

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Does it need to be multi-region?
Core to mission
• Can we be successful
without this service?
How much redundancy
do we need?
• How does this service
access persistent
storage?
Multi-Region
DNS routing
Single Region
Scheduled
Replication
Master/
Master
Near Real-Time
Eventual Consistency
from Master Copy
Master/
Replica
Read-Only Read-Write

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deciding multi-region is not black or white

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
multiRegionArchitecture
!=
multiRegionArchitecture
!=
multiRegionActiveActive
multiRegionArchitecture

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Business continuity
How much data can you afford to
recreate or lose?
How quickly must you recover?
What is the cost of downtime?
Disaster
Recovery point (RPO) Recovery time (RTO)
Data loss Down time
mission

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The four strategies for business continuity
Backup &
Restore
Pilot
Light
Warm
Standby
Active-
Active
AWS Multi-Region

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Strategy: Backup & restore (multi-region)
us-west-2
ap-southeast-1
App2
Server
Database
Server
Backup
Server
Backup to another region
• Use managed database services with
Amazon Simple Storage Service
(Amazon S3) or Amazon Glacier
• Data stored with high durability in multiple
locations
App1
Server
App3
Server
Data loss (RPO)
Down time (RTO)

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Strategy: Pilot light (multi-region)
ap-southeast-1
Web
Server
App1
Server
Database
Master
us-west-2
App2
Server
App3
Server
Database
Replica
App2
Server
Data loss (RPO)
Down time (RTO)
Database
Synchronization
Snapshots
Synchronization
Allows the scaling of redundant
sites during a failure scenario
Web
Server
App2
Server
Snapshots
AMIs:
Web, App, Database
Snapshots
AMIs:
Web, App, Database

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Strategy: Pilot light (multi-region)
ap-southeast-1
Web
Server
App1
Server
Database
Master
us-west-2
App2
Server
App3
Server
Database
Master
App2
Server
Data loss (RPO)
Down time (RTO)
Allows the scaling of redundant
sites during a failure scenario
X
Web
Server
App2
Server
Snapshots
AMIs:
Web, App, Database
Snapshots
AMIs:
Web, App, Database

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Strategy: Warm Standby (Multi-Region)
ap-southeast-1
Web
Server
App1
Server
Database
Master
us-west-2
App2
Server
App3
Server
Web
Server
App1
Server
Database
Replica
App2
Server
App3
Server
Data loss (RPO)
Down time (RTO)
Database
Synchronization
Snapshots
Synchronization
Snapshots
AMIs:
Web, App, Database
Snapshots
AMIs:
Web, App, Database

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Strategy: Warm standby (multi-region)
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
Data loss (RPO)
Down time (RTO)
X
us-west-2 ap-southeast-1
Snapshots
AMIs:
Web, App, Database
Snapshots
AMIs:
Web, App, Database

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pattern 1: Read local, write global
Snapshots
AMIs:
Web, App, Database
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
Web
Server
App1
Server
Database
Replica
App2
Server
App3
Server
us-west-2 ap-southeast-1
Snapshots
AMIs:
Web, App, Database
Users in
San Francisco
Users in
Taipeiread read& write
write
Snapshots
Synchronization
Database
Synchronization

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pattern 2: Read local, write partitioned
Snapshots
AMIs:
Web, App, Database
Web
Server
App1
Server
Database
Replica
App2
Server
App3
Server
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
us-west-2 ap-southeast-1
Snapshots
AMIs:
Web, App, Database
Taipei User
read & write
Snapshots
Synchronization
Database
Synchronization
visits Los Angeles
Taipei User
@ homeread
write
(shard @ app layer)
write
(shard @ app layer)

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pattern 3: Read local, write local
Snapshots
AMIs:
Web, App, Database
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
us-west-2 ap-southeast-1
Snapshots
AMIs:
Web, App, Database
Users in
San Francisco
Users in
Taipeiread read
Database
Synchronization
Multi-master, multi-region

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pattern 3: Read local, write local
Snapshots
AMIs:
Web, App, Database
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
us-west-2 ap-southeast-1
Snapshots
AMIs:
Web, App, Database
Users in
San Francisco
Users in
Taipeiread read& write
Database
Synchronization
.04 .03
& write
Multi-master, multi-region
Updated Object

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-region active-active patterns
Read local, write global
Read local, write partitioned
Read local, write local
Snapshots
AMIs:
Web, App, Database
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
Web
Server
App1
Server
Database
Replica
App2
Server
App3
Server
us-west-2 ap-southeast-1
Snapshots
AMIs:
Web, App, Database
Users in
San Francisco
Users in
Taipeiread read& write
write
Snapshots
Synchronization
Database
Synchronization
Snapshots
AMIs:
Web, App, Database
Web
Server
App1
Server
Database
Replica
App2
Server
App3
Server
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
us-west-2 ap-southeast-1
Snapshots
AMIs:
Web, App, Database
Taipei User
read & write
Snapshots
Synchronization
Database
Synchronization
visits Los Angeles
Taipei User
@ homeread
write
(shard @ app layer)
write
(shard @ app layer)
Snapshots
AMIs:
Web, App, Database
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
Web
Server
App1
Server
Database
Master
App2
Server
App3
Server
us-west-2 ap-southeast-1
Snapshots
AMIs:
Web, App, Database
Users in
San Francisco
Users in
Taipeiread read& write
Database
Synchronization
.04 .03
& write
Updated Object

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon S3 Availability Zones
S3 stores data in at least 3
Availability Zones (AZ’s)
Each AZ can be up to 8
physical data centers
Unavailability of a data center
or an AZ does not impact
overall S3 availability
Low latency private network
connect data centers and
AZ’s
Physically separate – even
extremely uncommon disasters
would only affect a single AZ
Data is automatically distributed
across a minimum of 3 AZ’s GEO
separated within an AWS Region

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 cross-region replication
Automatically replicate data to any other AWS Region
• Replicate by object, bucket, or prefix
• Support for server-side encryption on AWS Key Management Service
(AWS KMS) encrypted objects
• Ownership overwrite
•Change the object owner in the destination region
Region A Region B
Cross-region connectivity

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 cross-region replication examples
S3 Standard S3 Standard S3 Standard S-IA
S3 Standard Amazon Glacier
Zero-day lifecycle policy
to Amazon Glacier

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elastic Block Store (Amazon EBS) snapshots
Elastic block storage
• Point-in-time backup of modified volume
blocks
• Stored in S3, accessed via Amazon EBS
APIs
• Subsequent snapshots are incremental
• Deleting snapshot will only remove data
exclusive to that snapshot
• Copies in same region or cross region
EBS volume
EBS snapshot

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Build high performance, globally distributed applications
Low latency reads & writes to locally available tables
Disaster proof with multi-region redundancy
Easy to setup and no application re-writes required
Amazon DynamoDB global tables
Fully managed, multi-master, multi-region database

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-region read replicas with Amazon Relational
Database Service (Amazon RDS) and Amazon Aurora
• Horizontal scaling of read heavy workloads
• Offload long reporting queries
• Easy to re-create if fallen behind
• Cross-region read replicas bring data close to your applications in different regions
• Promote read replica to a master for faster recovery in the event of disaster

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inter-Region Virtual Private Cloud (VPC) PeeringInter-Region Virtual Private Cloud (VPC) Peering
Why is this important to my
architecture?

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon VPC + VPN
App1
Server
Database
App2
Server
App3
Server
App1
Server
Database
App2
Server
App3
Server
us-west-2 ap-southeast-1
VPC VPC
Software VPN
appliance
Software VPN
appliance
Internet
Gateway (IGW)
Internet
Gateway (IGW)

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inter-Region VPC Peering
App1
Server
Database
App2
Server
App3
Server
App1
Server
Database
App2
Server
App3
Server
us-west-2 ap-southeast-1
VPC VPC
AMAZON BACKBONE
VPC PEER

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Some notes…
Inter-Region VPC Peering encrypts with no single point of failure
or bandwidth bottleneck
Traffic using Inter-Region VPC Peering always stays on the
global AWS backbone
Inter-Region VPC Peering shares resources between regions or
replicates data for geographic redundancy

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-VPC connectivity
Regions can have multiple VPCs …
connected?

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inter-Region VPC Peering
App1
Server
Database
App2
Server
App3
Server
App1
Server
Database
App2
Server
App3
Server
us-west-2 ap-southeast-1
VPC VPC
AMAZON BACKBONE
VPC PEER

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inter-Region VPC Peering
us-west-2 ap-southeast-1
VPC
Shared
Services
AMAZON BACKBONE
VPC PEER
VPC Shared
Services

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-region multi-VPC connectivity
VPC
VPC
VPC
VPC
VPC
VPC
AMAZON BACKBONE
VPC PEER
Shared
Services
us-west-2 ap-southeast-1
VPC
VPC
VPC
VPC Shared
Services

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Route 53
Why is this important to my
architecture?

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traffic routing with Route 53
Latency-based routing
Amazon
Route53
Resource A
Resource B
* Latency numbers are only examples

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traffic routing with Route 53
Latency-based routing
Geolocation routing
Amazon
Route53
Resource A
In US
Resource B
in EU
User in US

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Traffic routing with Route 53
Latency-based routing
Geolocation routing
DNS failover
Amazon
Route53
Resource A
In US
Resource B
in EU
User in US

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Network Load Balancer (NLB)
Why is this important to my
architecture?

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
NLB + Inter-Region VPC Peering
VPC
VPC
Shared
Services
Shared
Services
us-west-2 ap-southeast-1
• Access NLBs over an inter-region peered VPC
• Load balance to IP-based targets deployed in
an inter-region peered VPC
• Support connections from clients to IP-based
targets in peered VPCs across different
Regions
• Route 53 health checks allow for traffic to be
shifted away from failed instances or regions

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Some notes…
Route 53 health checks and routes traffic based on routing policy
Multi-Region Multi-VPC Connectivity encrypts and shares
resources across the global AWS backbone
NLB can load balances across inter-region peered VPCs and
integrates with Route 53 health checks

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS CloudFormation StackSets
Admin Account
StackSet
Target
account A
Stack
Region 1
Account C Account D Account E …
Target
account B
Stack
Target
account A
Stack
Region 2
Account C Account D Account E …
Target
account B
Stack
Provision resources across multiple
AWS accounts and regions
https://amzn.to/2tNVHQl

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Config rules
Multi-account, multi-region data aggregation
• Integrates with AWS Organizations
• Evaluates configuration over time against policies defined using
AWS Config rules
• Alerts if configuration is noncompliant with your policies
Changing resources
AWS Config rules API access
Normalized
History, snapshot
Notifications

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Systems Manager (SM) Automation
• Take actions on AWS
resources centrally
Multi-region and multi-account actions
Account ID:
123456789
AWS Resources
N. Virginia
Account ID:
abc123def456
AWS Resources
Ireland
Account ID:
ab12345678de
AWS Resources
Mumbai• Remediate compliance
drifts
• Backup key resources

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inventory
Data
AWS Glue
Amazon Athena
Systems
Manager
Inventory
Systems Manager
Inventory
RegionARegionB
Account 1 Account 2
Resource Data Sync
Resource
Data Sync
Resource
Data Sync
SM Inventory
Detail View
AWS Systems Manager (SM) inventory
Multi-region and multi-account view
VPC
VPC
Systems Manager
Inventory
VPC

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Centralize Amazon CloudWatch Logs
Amazon
Kinesis Data Firehose
Amazon
S3
Amazon
Athena
Amazon
Elasticsearch Service
https://amzn.to/2kcMew1
AWS
Lambda
ap-southeast-1
AWS
Lambda
sa-east-1

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-region management
Cross accounts and multi-region
AWS CloudFormation
StackSets
Provision/Configure
AWS Config
Data Aggregation
Track resource
changes
AWS Systems Manager
Automation & Inventory
Perform operational
management
Amazon CloudWatch
Monitor and then
centralize logging

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Testing failure modes

60. ARC209
Christopher Lane
Enterprise Architect

61. Who is Chick-fil-A?
Chick-fil-A is a privately owned quick service restaurant
(QSR) company based in Atlanta, GA
2200+ restaurants in US (and opening in Canada soon!)
Highest per restaurant sales in QSR industry
More than 10% of revenue flowing through digital
channels

62. Aug 1
Redesigned app and national giveaway
No. 1 in App
Store!

63. Aug 30– Sep 29Aug 1
Redesigned app and national giveaway
Start of
giveaway

64. Aug 30– Sep 29Aug 1
Redesigned app and national giveaway
Doubled volume!
Start of
giveaway

65. Current architecture
Multi-region, active-passive
us-eastus-west
Critical
Service
s
Chick-fil-A
Services
Public
Private Private
Identity Identity
VPC
Peering
MasterReplica

66. Current architecture
Multi-region, active-passive
us-east
Public
Service
Tables
Chick-fil-A
Services

67. Current architecture
Multi-region, active-passive
us-east
Public
Services
Identity
Private
Master

68. Current architecture
Multi-region, active-passive
us-east
Public
Services
Identity
Private
us-west
Global
Tables!

69. Current architecture
Multi-region, active-passive
us-east
Public
Services
Identity
Private
Master
us-west
Replica

70. Current architecture
Multi-region, active-passive
us-east
Public
Services
Identity
Private
Master
us-west
Replica
Identity
VPC
Peering

71. Current architecture
Multi-region, active-passive
us-east
Public
Services
Identity
Private
Master
us-west
Replica
Identity
VPC
Peering
Critical
Service
s
% manually set

72. Future architecture
Multi-region, active-active +
Amazon Elastic Container Service for Kubernetes (Amazon EKS)
us-east
Public
Master
us-west
Master
Chick-fil-A
Services
Chick-fil-A
Services

73. Future architecture
Multi-region, active-active + Amazon EKS
us-east
Public
Master
us-west
Master
% based on health,
location
Chick-fil-A
Services
Chick-fil-A
Services

74. Routing tier
Profile
Service
Loyalty
Service
Ambassad
or
alb-ingress-
controller
ClusterIP
Service
/*
ALB
Ingress
Profile
Profile
Loyalt
y
Loyalt
y
/*

75. Routing tier
Profile
Profilealb-ingress-
controller
/profile
Ambassador
Profile
Service
Loyalty
Service
Loyalty
Loyalty
/* /*

76. Routing tier
Loyalty
Loyaltyalb-ingress-
controller
/loyalty
Ambassador
Profile
Service
Loyalty
Service
Profile
Profile
/* /*

77. “No goal is too high if you
climb with care and
confidence.”
Truett Cathy
medium.com/@cfatechblog
github.com/chick-fil-a
bit.ly/chickfilacareers
linkedin.com/in/lanecm

78. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
• Multi-region active-active architectures add significant complexity to
your stack – think carefully about your services and whether they need
to be multi-region at all.
• Consider whether less complex multi-region architectures, such as
multi-region backup, pilot light, or warm standby can meet your
needs.
• Design to avoid race conditions, by using read local/write global or
partitioned write.

79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
• Use storage tools to keep data synchronized, including S3 cross region
replication and EBS snapshot cross-region copies.
• Keeping data consistent across regions is challenging. Use DynamoDB
Global Tables or Read Replicas with RDS and Aurora.
• VPC peering allows consistent security across regions.
• Route 53 can perform failover checks, and most operations can be
automated – but the service still needs to be monitored. Create relevant
metrics and set alarms on them.

80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
• Plan to manage the environment! Use AWS CloudFormation StackSets,
AWS Config rules, AWS System Manager, and other DevOps tools.
• If you don’t test, it won’t work in a crisis. Test a lot.
• Eat mor chickin’.

81. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related sessions
ARC315 - Hands-On: Building a Multi-Region Active-Active Solution
ARC316 - Hands-On: Breaking a Multi-Region Active-Active Solution
ARC402 - Building Multi-Region Microservices
ARC415 – Building Multi-Region Persistence with MySQL
SRV214 - Multi-Region REST APIs with Automatic Failover
SRV326 - Build a Multi-Region Serverless Application for Resilience & High Availability
SRV425 - Best Practices for Building Multi-Region, Active-Active Serverless Applications

83. Time: 15 minutes after this session
Location: Speaker Lounge (ARIA East, Level 1, Willow Lounge)
Duration: 30 min.

84. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amy Che
Darin Briskman
Christopher Lane