HyperLogLog probabilistic data structures make some potent big data analytics magic. Making HLL work requires hard math. Understanding why it works does not.

1. HyperLogLog Intuition
(without the hard math)
Sim Simeonov, Founder & CTO, Swoop
@simeons / sim at swoop dot com

2. The following presentation takes a poetic
license in order to provide intuition

3. Q:How do we quickly count the number of
distinct things in some collection?
A: Since “things” is fuzzy, hash them to
simplify the problem to…

4. Q:How do we quickly determine the
cardinality (size) of a set of n numbers?
A: Quickly means using fewer resources.
Assume we only have k buckets…

5. Distribute n items randomly in k buckets
E(distance) ≅
!
"
E(min) ≅
!
"
⇒ 𝑛 ≅
!
#(%&")
more buckets == greater precision

6. We can estimate n from k and
the position of the first bucket…
without keeping any of the n numbers

7. Q:How do we improve the precision
of our estimate?
A: Use a collection of buckets and use the
mean of the estimates created from each.

8. HLL sketch ≅ a distribution of mins
true mean

9. HyperLogLog sketches are reaggregatable
because min reaggregates with min

10. Making it work in the real world
• Data is not uniformly distributed…
• Hash it!
• How do we get many “samples” from one set of hashes?
• Partition them!
• Can we get a good estimate for the mean?
• Yes, with some fancy math & empirical corrections.
• Do we actually have to keep the minimums?
• No, just keep the number of 0s before the first 1 in binary form.
https://research.neustar.biz/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/

11. http://bit.ly/spark-alchemy

12. http://bit.ly/spark-records

13. Improving patient outcomes
LEADING HEALTH DATA LEADING CONSUMER DATA
Lifestyle
Magazinesubscriptions
Catalogpurchases
Psychographics
Animal lover
Fisherman
Demographics
Propertyrecords
Internettransactions
• 280M unique US patients
• 7 years longitudinal data
• De-identified, HIPAA-safe
1st Party Data
Proprietary tech to
integrate data
NPI Data
Attributed to the
patient
Claims
ICD 9 or 10, CPT,
Rx and J codes
• 300M US Consumers
• 3,500+ consumer attributes
• De-identified, privacy-safe
Petabyte scale privacy-preserving ML/AI

14. • Experiment with the HLL functions in spark-alchemy.
• Keep big data in Spark only and interop with HLL sketches.
Do you want to make Spark great while improving millions of lives?
Let’s talk.
Calls to Action
sim at swoop dot com