At Airbnb, we use Redis extensively as an in-memory data store to reduce latency and provide sub-millisecond response for our website, search, images, payments, and more. We migrated our self-managed Redis environment from EC2 classic to fully-managed Amazon ElastiCache for Redis to reduce operational overhead and improve availability. Now, all our Redis is in an AWS managed service that provides multi-Availability Zone support, automatic failover, and maintenance. Attend this session to learn how we migrated our Redis environment while ensuring data integrity and zero downtime.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Airbnb’s Journey from Self-Managed
Redis to ElastiCache for Redis
Julie Trias
Site Reliability
Airbnb/Production Platform
D A T 3 1 9

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
About Airbnb
Redis at Airbnb
Why ElastiCache
Migration requirements
Migration procedure
Recap & take aways

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
About Airbnb

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is Airbnb?
• Founded in 2008
• Airbed and breakfast

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mission
To create a world where people can belong through healthy travel that is local,
authentic, diverse, inclusive, and sustainable.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Homes
5+ million
Countries
81,000
Cities
191
Unique homes

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Experiences
15,000+ 1,000+
MarketsUnique activities

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data stored
104 PB
Redis
940 TB
Aurora
237 TB
S3

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
History of Redis at Airbnb

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redis stats
Pre-migration
146 ElastiCache Redis
clusters
Post-migration
113 ElastiCache Redis
clusters
Today
82 EC2 Redis clusters

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use cases
Verification & fraud detection
Payments
Host and guest identity
Booking payments and host payments
Feeds
Homes and experiences feed
Messaging
Authentication
User login sessions
Airbnb app messaging service, as well as email and SMS
notifications

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SmartStack Redis
Pros
• Simple design
• Multi-AZ

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SmartStack Redis
Cons
• Operational overhead
• Manual failover
• No support for sharding
• No encryption support

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HARedis
Pros
• Auto failover
• Multi-AZ

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HARedis
Cons
• More operational overhead
• No support for sharding
• No encryption support
• Many restricted commands

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why ElastiCache?

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current approaches don’t work
Doesn’t scale
Difficult to
manage
Poor
experience Expensive

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ElastiCache Redis
Fully managed & hardened
Secure & compliant
AWS manages hardware, software, setup, configuration,
monitoring, failure recovery, and backups
VPC for cluster isolation, encryption at rest/transit,
HIPAA compliance
Highly available & reliable
Read replicas, multiple primaries, multi-AZ with
automatic failover
Easily scalable
Cluster with up to 6.1 TiB of in-memory data
Read scaling with replicas
Write and memory scaling with sharding
Scale out or in
Key-Value store
Fast in-memory data store in the cloud. Use as a database, cache,
message broker, queue

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration requirements

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Redis migration requirements
No cold startsZero
downtime
No data loss
No service interruption Data migration Dual-writes

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration procedure

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
High-level migration procedure
2. Deploy ElastiCache Redis cluster
Provision new cluster
4. Failover
Perform the switch
5. Cleanup
1. Stage the application
Prepare to redirect Redis traffic
Terminate EC2 Redis
3. Sync data
Start replication process

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stage the application
Be ready to push an update to production that
will redirect Redis traffic to ElastiCache

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New ElastiCache Redis cluster
• Single node cluster
• Multi-AZ failover disabled
• Allow traffic from EC2 Redis and application

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ElastiCache Redis
module "payments" {
source = "../module"
replication_group_id = “payments"
replication_group_description = ”Payments service."
node_type = "cache.r4.xlarge"
replica_count = 0
auto_failover_enabled = false
}

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sync data
• Amazon ElastiCache team enables slaveof
command
• Add the ElastiCache cluster as a replica in the
EC2 Redis cluster

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sync data
EC2-Redis-Primary:~ julie_trias$ $ redis-cli -h private_ip_of_ElastiCache_RG slaveof
private_ip_of_EC2_Redis port_of_EC2_Redis

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failover
Disable
alerts
Block
traffic
Allow sync
to finish
Block traffic to EC2 Redis

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failover
Update the application configuration to point
to the new ElastiCache Redis endpoint

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failover
EC2-Redis-Primary:~ julie_trias$ sudo service nerve stop
EC2-Redis-Primary:~ julie_trias$ redis-cli –p 5020 info replication
# Replication
role:master
connected_replicas:2
replica0:ip=10.0.1.109,port=6379,state=online,offset=760825652934,lag=0
replica1:ip=10.0.5.27,port=6379,state=online,offset=760825652934,lag=0

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failover
Update the application configuration to point
to the new ElastiCache Redis endpoint

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failover
Update the application configuration to point
to the new ElastiCache Redis endpoint

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ElastiCache Redis
EC2-Redis-Primary:~ julie_trias$ redis-cli –h
<replication_group_id>.dx6mdd.ng.0001.use1.cache.amazonaws.com slaveof no one

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failover
Promote Add replica
Enable multi-
AZ failover

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ElastiCache Redis
module "payments" {
source = "../module"
replication_group_id = “payments"
replication_group_description = ”Payments service."
node_type = "cache.r4.xlarge"
replica_count = 0
auto_failover_enabled = false
}

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ElastiCache Redis
module "payments" {
source = "../module"
replication_group_id = “payments"
replication_group_description = ”Payments service."
node_type = "cache.r4.xlarge"
auto_failover_enabled = false
}

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ElastiCache Redis
module "payments" {
source = "../module"
replication_group_id = “payments"
replication_group_description = ”Payments service."
node_type = "cache.r4.xlarge"
}

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failover
A fully-configured and loaded Redis

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cleanup
• Terminate EC2 Redis cluster instances
• Terminate proxy instances
• Delete Redis and proxy configurations everywhere

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recap & takeaways

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Recap
2. Deploy ElastiCache Redis cluster
Provision new cluster
4. Failover
Perform the switch
5. Cleanup
1. Stage the application
Prepare to redirect Redis traffic
Terminate EC2 Redis
3. Sync Data
Start replication process

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Takeaways
Fully managed & hardened
Secure & compliant
AWS manages hardware, software, setup, configuration,
monitoring, failure recovery, and backups
VPC for cluster isolation, encryption at rest/transit,
HIPAA compliance
Highly available & Reliable
Read replicas, multiple primaries, multi-AZ with
automatic failover
Easily scalable
Cluster with up to 6.1 TiB of in-memory data
Read scaling with replicas
Write and memory scaling with sharding
Scale out or in
Key-Value store
Fast in-memory data store in the cloud. Use as a database, cache,
message broker, queue

44. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Julie Trias
julie.trias@airbnb.com

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.