Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1OKo5FN. Danny Yuan discusses how stream processing is used in Uber's real-time system to solve a wide range of problems, including but not limited to real-time aggregation and prediction on geospatial time series, data migration, monitoring and alerting, and extracting patterns from data streams. Yuan also presents the architecture of the stream processing pipeline. Filmed at qconsf.com. Danny Yuan is a software engineer in Uber. He's currently working on streaming systems for Uber's logistics platform. Prior to joining Uber, he worked on building Netflix's cloud platform. His work includes predictive autoscaling, distributed tracing service, real-time data pipeline that scaled to process hundreds of billions of events every day, and Netflix's low-latency crypto services.

1. STREAM
PROCESSING@
UBERDANNY YUAN @ UBER

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/uber-stream-processing

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. What is Uber

5. Transportation at your fingertips

7. Stream Data Allows Us To Feel The Pulse Of Cities

8. Marketplace Health

9. What’s Going on Now

10. What’s Happened?

11. Status Tracking

15. A Little Background

16. Uber’s Platform Is a Distributed State Machine
Rider States

17. Uber’s Platform Is a Distributed State Machine
Rider States Driver States

18. Applications can’t do everything

19. Instead, Applications Emit Events

20. Events Should Be Available In Seconds

21. Events Should Rarely Get Lost

22. Events Should Be Cheap And Scalable

24. Where are the challenges?

25. Many Dimensions
Dozens of fields per event

26. Granular Data

27. Granular Data

28. Granular Data
Over 10,000 hexagons in the city

29. Granular Data
7 vehicle types

30. Granular Data
1440 minutes in a day

31. Granular Data
13 driver states

32. Granular Data
300 cities

33. Granular Data
1 day of data: 300 x 10,000 x 7 x 1440 x 13 = 393 billion
possible combinations

34. Unknown Query Patterns
Any combination of dimensions

35. Variety of Aggregations
- Heatmap
- Top N
- Histogram
- count(), avg(), sum(), percent(), geo

36. Different Geo Aggregation

37. Large Data Volume
• Hundreds of thousands of events per
second, or billions of events per day
• At least dozens of fields in each event

38. Tight Schedule

39. Key: Generalization

40. Data Type
• Dimensional Temporal Spatial Data
Dimension Value
state driver_arrived
vehicle type uber X
timestamp 13244323342
lattitude 12.23
longitude 30.00

41. Data Query
• OLAP on single-table temporal-spatial data
SELECT
<agg
functions>,
<dimensions>
FROM
<data_source>
WHERE
<boolean
filter>
GROUP
BY
<dimensions>
HAVING
<boolean
filter>
ORDER
BY
<sorting
criterial>
LIMIT
<n>
DO
<post
aggregation>

42. Finding the Right Storage System

43. Minimum Requirements
• OLAP with geospatial and time series support
• Support large amount of data
• Sub-second response time
• Query of raw data

44. It can’t be a KV store

45. Challenges to KV Store
Pre-computing all keys is O(2n)
for both space
and time

46. It can’t be a relational database

47. Challenges to Relational DB
• Managing multiple indices is painful
• Scanning is not fast enough

48. A System That Supports
• Fast scan
• Arbitrary boolean queries
• Raw data
• Wide range of aggregations

49. Elasticsearch

50. Highly Efficient Inverted-Index For Boolean Query

51. Built-in Distributed Query

52. Fast Scan with Flexible Aggregations

53. Storage

54. Are We Done?

55. Transformation
e.g. (Lat, Long) -> (zipcode, hexagon)

56. Dynamic Pricing

57. Trend Prediction

58. Supply and Demand Distribution

59. Technically Speaking: Clustering & Pr(D, S, E)

60. New Use Cases —> New Requirements

61. Pre-aggregation

62. Joining Multiple Streams

63. Sessionization

64. Multi-Staged Processing

65. State Management

66. Apache Samza

67. Why Apache Samza?

68. DAG on Kafka

69. Excellent Integration with Kafka

70. Excellent Integration with Kafka

71. Built-in Checkpointing

72. Built-in State Management

73. Processing Storage

74. What If Storage Is Down?

75. What If Processing Takes Long?

76. Processing Storage

77. Are We Done?

80. Post Processing

81. Results Transformation and Smoothing

82. Scale of Post Processing
10,000 hexagons in a city

83. Scale of Post Processing
331 neighboring hexagons to look at

84. Scale of Post Processing
331 x 10,000 = 3.1 Million Hexagons to
Process for a Single Query

85. Scale of Post Processing
99%-ile Processing Time: 70ms

86. Post Processing
• Each processor is a pure function
• Processors can be composed by combinators

87. Post Processing
• Highly parallelized execution
• Pipelining

88. Post Processing
• Each processor is a pure function
• Processors can be composed by combinators
• Highly parallelized execution

89. Practical Considerations

90. Data Discovery

91. Elasticsearch Query Can Be Complex

92. /driverAcceptanceRate?
geo_dist(10,
[37,
22])&
time_range(2015-­‐02-­‐04,2015-­‐03-­‐06)&
aggregate(timeseries(7d))&
eq(msg.driverId,1)

93. Elasticsearch Query Can Be Optimized
• Pipelining
• Validation
• Throttling

94. Timeinseconds

95. Elasticsearch Can Be Replaced

96. Storage QueryProcessing

97. There’s one more thing

98. There are always patterns in streams

99. There is always need for quick exploration

100. How many drivers cancel a request 10 times in a
row within a 5-minute window?

101. Which riders request a pickup from 100 miles
apart within a half hour window?

103. Complex Event Processing
FROM
driver_canceled#window.time(10
min)
SELECT
clientUUID,
count(clientUUID)
as
cancelCount
GROUP
BY
clientUUID
HAVING
cancelCount
>
10
INSERT
INTO
hipchat(room);

104. Implementation Becomes Easy

105. Thank You!

106. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/uber-
stream-processing