Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1TJPw40. Fangjin Yang covers common problems and failures seen with distributed systems, and discusses design patterns that can be used to maintain data integrity and availability when everything goes wrong. Yang uses Druid as a real world case study of how these patterns are implemented in an open source technology. Filmed at qconsf.com. Fangjin Yang is one of the main committers to the open source Druid project and a co-founder of Imply, a San Francisco based data startup.

1. Architecting Distributed Databases for
Failure
A Case Study with Druid
Fangjin Yang
Cofounder @ Imply

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations/
data-integrity-distributed-systems

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Overview
The Bad
The Really Bad
The Catastrophic
Best Practices: Operations

5. Everything is going to fail!

6. Requirements
Scalable
- Tens of thousands of nodes
- Petabytes of raw data
Available
- 24 x 7 x 365 uptime
Performant
- Run as smoothly as possible when things go wrong

7. Druid
Open source distributed data store
Column oriented storage of event data
Low latency OLAP queries & low latency data ingestion
Initially designed to power a SaaS for online advertising (in AWS)
Our real-world example case study

8. The Bad

9. Single Server Failures
Common
Occurs for every imaginable and unimaginable reason
- Hardware malfunction, kernel panic, network outage, etc.
- Minimal impact
Standard solution: replication

10. Druid Segments
Timestamp Dimensions Measures
2015-01-01T00
2015-01-01T01
2015-01-02T05
2015-01-02T07
2015-01-03T05
2015-01-03T07
Timestamp Dimensions Measures
2015-01-01T00
2015-01-01T01
Timestamp Dimensions Measures
2015-01-02T05
2015-01-02T07
Timestamp Dimensions Measures
2015-01-03T05
2015-01-03T07
Partition by time
Segment_2015-01-01/2014-01-02
Segment_2015-01-02/2014-01-03
Segment_2015-01-03/2014-01-04

11. Replication Example
Segment_2015-01-01/2015-01-02
(Segment_1)
Segment_2015-01-01/2015-01-02
(Segment_2)
Segment_2015-01-01/2015-01-02
(Segment_3)
Segment_1
Segment_2
Segment_1
Segment_3
Segment_2
Segment_3
Load
Druid Historicals
Druid BrokersQueries
Client

12. Query Segment_1
Segment_2015-01-01/2015-01-02
(Segment_1)
Segment_2015-01-01/2015-01-02
(Segment_2)
Segment_2015-01-01/2015-01-02
(Segment_3)
Segment_1
Segment_2
Segment_1
Segment_3
Segment_2
Segment_3
Load
Druid Historicals
Druid BrokersQueries
Client

13. Query Segment_1
Segment_2015-01-01/2015-01-02
(Segment_1)
Segment_2015-01-01/2015-01-02
(Segment_2)
Segment_2015-01-01/2015-01-02
(Segment_3)
Segment_1
Segment_2
Segment_1
Segment_3
Segment_2
Segment_3
Load
Druid Historicals
Druid BrokersQueries
Client

14. Multi-Server Failures
Common: 1 server fails
Less common: >1 server fails
Data center issues (rack failure)
Two strategies:
- fast recovery
- multi-datacenter replication

15. Fast Recovery
Complete data availability in the face of multi-server failures is hard!
Focus on fast recovery instead
Be careful of the pitfalls of fast recovery
More viable in the cloud

16. Fast Recovery Example
Segment_2015-01-01/2015-01-02
(Segment_1)
Segment_2015-01-01/2015-01-02
(Segment_2)
Segment_2015-01-01/2015-01-02
(Segment_3)
Segment_1
Segment_2
Segment_1
Segment_3
Segment_2
Segment_3
Load
Druid Historicals
Druid BrokersQueries
Client
Deep Storage
(S3/HDFS)
Load

17. Fast Recovery Example
Segment_1
Segment_2
Segment_1
Segment_3
Segment_2
Segment_3
Druid Historicals
Load
Deep Storage
(S3/HDFS)

18. Fast Recovery Example
Segment_1
Segment_2
Druid Historicals
Deep Storage
(S3/HDFS)
Load

19. Fast Recovery Example
Segment_1
Segment_2
Druid Historicals
Deep Storage
(S3/HDFS)
Load
Druid Coordinator
Load Segment_1,
Segment_3
Load Segment_2,
Segment_3

20. Fast Recovery Example
Segment_1
Segment_2
Druid Historicals
Deep Storage
(S3/HDFS)
Load
Druid Coordinator
Segment_1
Segment_3
Segment_2
Segment_3

21. Dangers of Fast Recovery
Easy to create bottlenecks
- Prioritize how resources are spent during recovery
- Druid prioritizes data availability and throttles replication
Beware query hotspots
- Intelligent load balancing during recovery is important

22. Fast Recovery Example
Segment_1
Segment_2
Segment_1
Segment_3
Segment_2
Segment_3
Druid Historicals
Load
Deep Storage
(S3/HDFS)

23. Fast Recovery Example
Druid Historicals
Deep Storage
(S3/HDFS)
Load
Segment_1
Segment_2
Segment_3
Overloaded!

24. The Really Bad

25. Data Center Outage
Very uncommon
Power loss
Can be extremely disruptive without proper planning
Solution: Multi-datacenter replication
Beware pitfalls of multi-datacenter replication

26. Multi-Datacenter Replication
Segment_1
Segment_2
Segment_3
Segment_1
Segment_3
Segment_2
Segment_3
Druid Historicals
Druid BrokersQueries
Client
Druid Coordinator
Data Center 1
Data Center 2

27. Multi-Datacenter Pitfalls
Coordination + leader election can be tricky
Communication can require non-trivial network time
Coordination usually done with heartbeats and quorum decisions
Writes, failovers, & consistent reads require round trips

28. Multi-Datacenter Replication
Client
Data Center 1
Data Center 2

29. The Catastrophic

30. “Why are things slow today?”
Poor performance is much worse than things completely failing
Causes:
- Heavy concurrent usage (multi-tenancy)
- Hotspots & variability
- Bad software update

31. Architecting for Multi-tenancy
Small units of computation
- No single query should starve out a cluster

32. Druid Multi-tenancy
Segment_query_1
Segment_query_2
Segment_query_1
Segment_query_3
Segment_query_2
Segment_query_1
Segment_query_4
Druid Historical
Queries
Processing
Order

33. Architecting for Multi-tenancy
Resource prioritization and isolation
- Not all queries are equal
- Not all users are equal

34. Druid Multi-tenancy
Tier 1: Older
Data
Tier 2: Newer
Data
Tier 2: Newer
Data
Dedicated
for Older
data
Druid Historicals
Druid Brokers
Dedicated
for Newer
Dat
Queries
Client

35. Hotspots
Incredible variability in query performance among nodes
Nodes may become slow but not fail
Difficult to detect as there is nothing obviously wrong
Solutions:
- Hedged requests
- Selective Replication
- Latency Induced Probation

36. Hedged Requests
Segment_1
Segment_2
Segment_1
Segment_3
Segment_2
Segment_3
Druid Historicals
Druid Brokers
Client

37. Hedged Requests
Segment_1
Segment_2
Segment_1
Segment_3
Segment_2
Segment_3
Druid Historicals
Druid Brokers
Client

38. Minimizing Variability
Selective Replication
Latency-induced probation
Great paper: https://web.stanford.edu/class/cs240/readings/tail-at-scale.pdf

39. Bad Software Updates
It is very difficult to simulate production traffic
- Testing/staging clusters mostly verify correctness
No noticeable failures for a long time
Common cause of cascading failures

40. Rolling Upgrades
Be able to update different components with no down time
Backwards compatibility is extremely important
Roll back if things are bad

41. Rolling Upgrades
V2
V1
V1
V1
Druid Historicals
Druid Brokers
V1
Queries
Client

42. Rolling Upgrades
V2
V2
V2
V1
Druid Historicals
Druid Brokers
V1
Queries
Client

43. Rolling Upgrades
V2
V2
V2
V2
Druid Historicals
Druid BrokersQueries
Client
V1

44. Best Practices: Operations

45. Monitoring
Detection of when things go badly
Define your critical metrics and acceptable values

46. Alerts
Alert on critical errors
- Out of disk space, out of cluster capacity, etc.
Design alerts to reduce “noise”
- Distinguish warnings and alerts

47. Exploratory Analytics
Extremely critical to diagnosing root causes quickly
Not many organizations do this

48. Takeaways
Everything is going to fail!
- Use replication for single server failures
- Use fast recovery for multi-server failures (when you don’t want to set up
another data center)
- Use multi-datacenter replication when availability really matters
- Alerting, monitoring, and exploratory analysis are critical

49. Thanks!
@implydata
@druidio
@fangjin
imply.io
druid.io

50. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/data-
integrity-distributed-systems