Data Streaming Algorithms

1. Streaming Algorithms
Eric Fu
1

2. Streaming algorithms have to be approximate
• Space cost <= O(logN)
• Time cost <= O(N)
• online

3. Problem Statement
In limited space, in one pass, over a sequence of items
Compute the following
min, max, average,
standard deviation
moving average
Cardinality (count of distinct items in a stream)
Heavy hitters (aka find most frequent items)
Order statistics (rank of an item in sorted sequence)
3

4. Key Tricks
• hashing
• sketching

5. Hyper log log
Cardinality (count of distinct items in a stream)

6. Repeated Minimum

7. Bits emitted by a hash
In hash of all items, observe number of times you get bit ‘1’ followed
by many zeros
7

8. Bit patterns
For num = [1, 1000]
h = hash(num)
Number of hashes ending in Out of 1000
0 530
10 281
100 140
1000 53
10000 28
100000 9
1000000 12
10000000 5
100000000 2
1000000000 0
10000000000 0
100000000000 0
8
Bit ‘1’ followed by 9 or
more zeroes not found
Because 1000 ~ 2^10

9. Flajolet-Martin sketch algo
1. For each item
2. Index = rightmost bit in hash(item)
3. Bitmap[index] = 1
(at this point, bitmap = “000...00000101011111”)
1. Estimated N ~ 2 rightmost ‘0’ bit in bitmap
9
Further improvements : split stream into M substreams and use harmonic mean of their
counters, use 64-bit hash instead of 32, add custom correction factors to hash at low and high
range.

10. Comparison between 3 different versions
* my FM-sketch implementation is incomplete – actual algo is not that bad
10
X : actual cardinality
Y : estimated
cardinality

11. What is a sketch ?
• A sketch maintains one or more “random variables”
which provide answers that are probabilistically
accurate.
• In Hyperloglog, this random variable is the “position
of the rightmost zero”. It roughly estimates the
actual cardinality of the set.
• A sketch uses universal hash function to distribute
data uniformly.
• To reduce variance, it may use many pairwise-
independent hashes and take their average.
11
* all random variables do not have
normal distribution. Above Pic is to
help in visualizing

12. MJRTY
Find majority

13. How would you determine the majority element of:
sequence: A A A C C B B C C C B C C
https://www.cs.utexas.edu/~moore/best-ideas/mjrty/index.html

14. Count-Min Sketch
Heavy hitters (find most frequent items)
14

15. Bloom Filter

16. Count-Min sketch
Insert Probe

17. Count-Min sketch
For heavy hitters, need additional heap data structure to maintain
those items which hashed to high value slots.

18. Leaky counters
Heavy hitters (find most frequent items)

19. Observation

20. Karp
1. Keep a frequency Map<item, count>
2. For each v in sequence
3. increment Map[v].count
4. If map.size() > threshold
5. for each element in Map
6. decrement Map[element].count
7. if count is zero, delete Map[element]
Algo has second pass to adjust counts. Paper discusses additional optimizations.
Implemented in Apache Spark. See DataFrameStatFunctions.freqItems().
Maintain a truncated histogram
20

21. Frugal Streaming
Order statistics (rank of an item in sorted sequence)
21

22. Order statistics terminology
Given sorted sequence [1, 1, 1, 2, 3]
1. 0-quantile = minimum
2. 0.25 quantile = 1st quartile = 25 percentile
3. 0.50 quantile = 2nd quartile = 50 percentile = median
4. 0.75 quantile = 3rd quartile = 75 percentile
5. 1-quantile = maximum
22

23. Order statistics offline algorithm
• There exists an offline and exact algorithm to find the kth item in a set
• QuickSelect (Blum, et al) which is effectively a truncated quicksort
• Can run in linear time algorithm (depending on pivot)
23

24. Frugal streaming
1. Median_est = 0
2. For v in stream
3. if (v > median_est)
4. Increment median_est
5. else if (v < median_est)
6. Decrement median_est
24
Memory = log(N) bits where N = cardinality
Caveat: Reported median may not be in the stream
Performs poorly on sorted data
Works best if stream items are independent and random
Median drift s in the direction of the true median.
Probability of drifting after reaching true median is low.
Paper discusses extension to compute other quantiles
4 2 1 5 52 43
4 4 2 4 33 43
2 1 2 32 43
Stream
True median
estimated 1

25. Frugal streaming (75%)
1. quantile_75 = 0
2. For v in stream
3. r = random()
4. if (v > quantile_75 && r > 1 – 0.75)
5. Increment quantile_75
6. else if (v < quantile_75 && r > 0.75)
7. Decrement quantile_75

26. Streaming k-means
Order statistics (rank of an item in sorted sequence)

27. K-means in ML

28. More Clusters

29. t-Digest - Dunning et al
29
Each centroid attracts points nearest to it. Keeps “average” and “count” of
these points.
Maintain a balanced binary tree of centroid nodes

30. t-Digest

31. t-Digest
• Use sorted structure to find quantiles.
• Centroids at both ends are deliberately kept small to increase accuracy of
outliers.
• Can merge two T-digests.
• Performs poorly on ascending/descending stream.
31

32. Thanks!