Presented at Data Day Texas on January 10th, 2015

1. Using simplicity to
make hard Big Data
problems easy

2. MasterMaster
DatasetDataset
Batch viewsBatch views
New DataNew Data
RealtimeRealtime
viewsviews
QueryQuery
Lambda Architecture

3. MasterMaster
DatasetDataset
R/WR/W
databasesdatabases
StreamStream
processorprocessor
Proposed alternative
(Problematic)

4. Easy problem

5. struct PageView {
UserID id,
String url,
Timestamp timestamp
}

6. Implement:
function NumUniqueVisitors(
String url,
int startHour,
int endHour)

7. Unique visitors over a
range of hours

8. Notes:
• Not limiting ourselves to current tooling
• Reasonable variations of existing tooling
are acceptable
• Interested in what’s fundamentally possible

9. Traditional
Architectures
ApplicationApplication DatabasesDatabases
ApplicationApplication DatabasesDatabases
StreamStream
processorprocessor
QueueQueue
Synchronous
Asynchronous

10. Approach #1
• Use Key->Set database
• Key = [URL, hour bucket]
• Value = Set of UserIDs

11. Approach #1
• Queries:
• Get all sets for all hours in range of
query
• Union sets together
• Compute count of merged set

12. Approach #1
• Lot of database lookups for large ranges
• Potentially a lot of items in sets, so lots of
work to merge/count
• Database will use a lot of space

13. Approach #2
Use HyperLogLog

14. interface HyperLogLog {
boolean add(Object o);
long size();
HyperLogLog merge(HyperLogLog... otherSets);
}

15. Approach #2
• Use Key->HyperLogLog database
• Key = [URL, hour bucket]
• Value = HyperLogLog structure

16. Approach #2
• Queries:
• Get all HyperLogLog structures for all
hours in range of query
• Merge structures together
• Retrieve count from merged structure

17. Approach #2
• Much more efficient use of storage
• Less work at query time
• Mild accuracy tradeoff

18. Approach #2
• Large ranges still require lots of database
lookups / work

19. Approach #3
• Use Key->HyperLogLog database
• Key = [URL, bucket, granularity]
• Value = HyperLogLog structure

20. Approach #3
• Queries:
• Compute minimal number of database
lookups to satisfy range
• Get all HyperLogLog structures in range
• Merge structures together
• Retrieve count from merged structure

21. Approach #3
• All benefits of #2
• Minimal number of lookups for any range,
so less variation in latency
• Minimal increase in storage
• Requires more work at write time

22. Hard problem

23. struct Equiv {
UserID id1,
UserID id2
}
struct PageView {
UserID id,
String url,
Timestamp timestamp
}

25. Person A Person B

26. Implement:
function NumUniqueVisitors(
String url,
int startHour,
int endHour)

27. [“foo.com/page1”, 0]
[“foo.com/page1”, 1]
[“foo.com/page1”, 2]
...
[“foo.com/page1”, 1002]
{A, B, C}
{B}
{A, C, D, E}
...
{A, B, C, F, Z}

28. [“foo.com/page1”, 0]
[“foo.com/page1”, 1]
[“foo.com/page1”, 2]
...
[“foo.com/page1”, 1002]
{A, B, C}
{B}
{A, C, D, E}
...
{A, B, C, F, Z}
A <-> C

29. Any single equiv could change any bucket

30. No way to take advantage of HyperLogLog

31. Approach #1
• [URL, hour] -> Set of PersonIDs
• UserID -> Set of buckets
• Indexes to incrementally normalize
UserIDs into PersonIDs

32. Approach #1
• Getting complicated
• Large indexes
• Operations require a lot of work

33. Approach #2
• [URL, bucket] -> Set of UserIDs
• Like Approach 1, incrementally normalize
UserId’s
• UserID -> PersonID

34. Approach #2
• Query:
• Retrieve all UserID sets for range
• Merge sets together
• Convert UserIDs -> PersonIDs to
produce new set
• Get count of new set

35. Incremental UserID
normalization

36. Attempt 1:
• Maintain index from UserID -> PersonID
• When receive A <-> B:
• Find what they’re each normalized to,
and transitively normalize all reachable
IDs to “smallest” val

37. 1 <-> 4
1 -> 1
4 -> 1
2 <-> 5
5 -> 2
2 -> 2
5 <-> 3 3 -> 2
4 <-> 5
5 -> 1
2 -> 1
3 -> 1 never gets produced!

38. Attempt 2:
• UserID -> PersonID
• PersonID -> Set of UserIDs
• When receive A <-> B
• Find what they’re each normalized to, and
choose one for both to be normalized to
• Update all UserID’s in both normalized sets

39. 1 <-> 4
1 -> 1
4 -> 1
1 -> {1, 4}
2 <-> 5
5 -> 2
2 -> 2
2 -> {2, 5}
5 <-> 3 3 -> 2
2 -> {2, 3, 5}
4 <-> 5
5 -> 1
2 -> 1
3 -> 1
1 -> {1, 2, 3, 4, 5}

40. Challenges
• Fault-tolerance / ensuring consistency
between indexes
• Concurrency challenges

41. General challenges with
traditional
architectures
• Redundant storage of information
(“denormalization”)
• Brittle to human error
• Operational challenges of enormous
installations of very complex databases

42. MasterMaster
DatasetDataset
Indexes forIndexes for
uniques overuniques over
timetime
StreamStream
processorprocessor
No fully incremental approach will work!

43. Let’s take a completely different approach!

44. Some Rough
Definitions
Complicated: lots of parts

45. Some Rough
Definitions
Complex: intertwinement between separate functions

46. Some Rough
Definitions
Simple: the opposite of complex

47. Real World Example

48. ID Name
Location
ID
1 Sally 3
2 George 1
3 Bob 3
Location
ID
City State Population
1 New York NY 8.2M
2 San Diego CA 1.3M
3 Chicago IL 2.7M
Normalized schema
Normalization vs
Denormalization

49. Join is too expensive, so
denormalize...

50. ID Name Location ID City State
1 Sally 3 Chicago IL
2 George 1 New York NY
3 Bob 3 Chicago IL
Location ID City State Population
1 New York NY 8.2M
2 San Diego CA 1.3M
3 Chicago IL 2.7M
Denormalized schema

51. Complexity between robust data model
and query performance

52. Allow queries to be out of date by hours

53. Store every Equiv and PageView
MasterMaster
DatasetDataset

54. MasterMaster
DatasetDataset
Continuously recompute indexes
Indexes forIndexes for
uniques overuniques over
timetime

55. Indexes = function(all data)

57. Iterative graph algorithm

58. Join

59. Basic aggregation

60. Sidenote on tooling
• Batch processing systems are tools to
implement function(all data) scalably
• Implementing this is easy

61. Person 1 Person 6
UserID normalization

62. UserID normalization

67. Conclusions
• Easy to understand and implement
• Scalable
• Concurrency / fault-tolerance easily
abstracted away from you
• Great query performance

68. Conclusions
• But... always out of date

69. Absorbed into batch viewsAbsorbed into batch views
NotNot
absorbedabsorbed
No
wTime
Just a small
percentage
of data!

70. MasterMaster
DatasetDataset
Batch viewsBatch views
New DataNew Data
RealtimeRealtime
viewsviews
QueryQuery

71. Get historical buckets from batch views
and recent buckets from realtime views

72. Implementing realtime
layer
• Isn’t this the exact same problem we faced
before we went down the path of batch
computation?

73. Approach #1
• Use the exact same approach as we did in
fully incremental implementation
• Query performance only degraded for
recent buckets
• e.g., “last month” range computes vast
majority of query from efficient batch
indexes

74. Approach #1
• Relatively small number of buckets in
realtime layer
• So not that much effect on storage costs

75. Approach #1
• Complexity of realtime layer is softened by
existence of batch layer
• Batch layer continuously overrides realtime
layer, so mistakes are auto-fixed

76. Approach #1
• Still going to be a lot of work to implement
this realtime layer
• Recent buckets with lots of uniques will
still cause bad query performance
• No way to apply recent equivs to batch
views without restructuring batch views

77. Approach #2
• Approximate!
• Ignore realtime equivs

78. UserID ->UserID ->
PersonIDPersonID
(from batch)
Approach #2
PageviewPageview
Convert UserIDConvert UserID
to PersonIDto PersonID
[URL, bucket][URL, bucket]
->->
HyperLogLogHyperLogLog

79. Approach #2
• Highly efficient
• Great performance
• Easy to implement

80. Approach #2
• Only inaccurate for recent equivs
• Intuitively, shouldn’t be that much
inaccuracy
• Should quantify additional error

81. Approach #2
• Extra inaccuracy is automatically weeded
out over time
• “Eventual accuracy”

82. Simplicity

83. Input:
Normalize/denormalize
Output:
Data model robustness
Query performance

84. MasterMaster
DatasetDataset
Batch viewsBatch views
Normalized
Robust data model
Denormalized
Optimized for queries

85. Normalization problem
solved
• Maintaining consistency in views easy
because defined as function(all data)
• Can recompute if anything ever goes
wrong

86. Human fault-tolerance

87. Complexity of
Read/Write Databases

88. Black box fallacyBlack box fallacy

89. Incremental compaction
• Databases write to write-ahead log before
modifying disk and memory indexes
• Need to occasionally compact the log and
indexes

90. Memory Disk
Write-ahead log

91. Memory Disk
Write-ahead log

92. Memory Disk
Write-ahead log
Compaction

93. Incremental compaction
• Notorious for causing huge, sudden
changes in performance
• Machines can seem locked up
• Necessitated by random writes
• Extremely complex to deal with

94. More Complexity
• Dealing with CAP / eventual consistency
• “Call Me Maybe” blog posts found data loss
problems in many popular databases
• Redis
• Cassandra
• ElasticSearch

95. MasterMaster
DatasetDataset
Batch viewsBatch views
New DataNew Data
RealtimeRealtime
viewsviews
QueryQuery

96. MasterMaster
DatasetDataset
Batch viewsBatch views
New DataNew Data
RealtimeRealtime
viewsviews
QueryQuery
No random writes!

97. MasterMaster
DatasetDataset
R/WR/W
databasesdatabases
StreamStream
processorprocessor

98. MasterMaster
DatasetDataset
ApplicationApplication
R/WR/W
databasesdatabases
(Synchronous version)

99. MasterMaster
DatasetDataset
Batch viewsBatch views
New DataNew Data
RealtimeRealtime
viewsviews
QueryQuery
Lambda Architecture

100. Lambda = Function
Query = Function(All Data)

101. Lambda Architecture
• This is most basic form of it
• Many variants of it incorporating more
and/or different kinds of layers