At Nielsen Identity, we leverage Druid to provide our customers with real-time analytics tools for various use-cases, including in-flight analytics, reporting and building target audiences. The common challenge of these use-cases is counting distinct elements in real-time at scale. We’ve been using Druid to solve these problems for the past 4 years, and gained a lot of experience with it. In this talk, we will share some of the best practices and tips we’ve gathered over the years, including: *Data modeling *Ingestion *Retention and deletion *Query optimization

1. Casting the Spell:
Druid in Practice
October 2020
Yakir Buskilla, Nielsen
Itai Yaffe, Imply
1

2. @ItaiYaffe, @yakiro

3. @ItaiYaffe, @yakiro
… …

4. @ItaiYaffe, @yakiro
… …

5. @ItaiYaffe, @yakiro
… …

6. Casting the Spell:
Druid in Practice
October 2020
Yakir Buskilla, Nielsen
Itai Yaffe, Imply
6

7. @ItaiYaffe, @yakiro
●
●
●
●
●

8. @ItaiYaffe, @yakiro
01. 02.
03. 04.

9. @ItaiYaffe, @yakiro
01. 02.
03. 04.

10. @ItaiYaffe, @yakiro
●
●
●
○
○

11. @ItaiYaffe, @yakiro
>10B events/day >30TB/day
S3
3000’s nodes/day
10’s of TB
ingested/day
druid

12. @ItaiYaffe, @yakiro

13. @ItaiYaffe, @yakiro
Awareness Consideration Intent Purchase
Drop-off
Drop-off
Drop-off

14. @ItaiYaffe, @yakiro

15. @ItaiYaffe, @yakiro
●

16. @ItaiYaffe, @yakiro
●
●
●
●

17. @ItaiYaffe, @yakiro
●
●
○
●
○
○
●

18. @ItaiYaffe, @yakiro

19. @ItaiYaffe, @yakiro

20. @ItaiYaffe, @yakiro

21. @ItaiYaffe, @yakiro

22. @ItaiYaffe, @yakiro

23. @ItaiYaffe, @yakiro

24. @ItaiYaffe, @yakiro

25. @ItaiYaffe, @yakiro

26. @ItaiYaffe, @yakiro

27. @ItaiYaffe, @yakiro
●
○
●
●

28. @ItaiYaffe, @yakiro

29. @ItaiYaffe, @yakiro
●
○
●
○

30. @ItaiYaffe, @yakiro
●
○
●
○

31. @ItaiYaffe, @yakiro
●
○
○
●

32. @ItaiYaffe, @yakiro
●
○
●
○
○
○

33. @ItaiYaffe, @yakiro
●
○
○

34. @ItaiYaffe, @yakiro
●
●
○
○

35. @ItaiYaffe, @yakiro

36. @ItaiYaffe, @yakiro
●
○
○
●
○
●
○

37. @ItaiYaffe, @yakiro
●
○
○
●
○

38. @ItaiYaffe, @yakiro
●
●

39. @ItaiYaffe, @yakiro
●
●
●

40. @ItaiYaffe, @yakiro
●
●
●

41. @ItaiYaffe, @yakiro

42. @ItaiYaffe, @yakiro

43. @ItaiYaffe, @yakiro
●
○
○
●

44. @ItaiYaffe, @yakiro
●
●
●

45. @ItaiYaffe, @yakiro

46. @ItaiYaffe, @yakiro
●
●

47. @ItaiYaffe, @yakiro
●
●

48. @ItaiYaffe, @yakiro
● …
●
○
○
○

49. @ItaiYaffe, @yakiro
●
○
○
■
■

50. @ItaiYaffe, @yakiro
●
○ SELECT APPROX_COUNT_DISTINCT_DS_THETA(user_id_sketch)
FROM campaign_1012
WHERE tactic = 1 AND
__time BETWEEN TIMESTAMP '2018-02-01' AND TIMESTAMP '2020-09-08'

51. @ItaiYaffe, @yakiro
●
○ SELECT APPROX_COUNT_DISTINCT_DS_THETA(user_id_sketch)
FROM campaign_1012
WHERE tactic = 1 AND
__time BETWEEN TIMESTAMP '2018-02-01' AND TIMESTAMP '2020-09-08'
○ SELECT APPROX_COUNT_DISTINCT_DS_THETA(user_id_sketch,65536)...

52. @ItaiYaffe, @yakiro
●
○ SELECT APPROX_COUNT_DISTINCT_DS_THETA(user_id_sketch)
FROM campaign_1012
WHERE tactic = 1 AND
__time BETWEEN TIMESTAMP '2018-02-01' AND TIMESTAMP '2020-09-08'
○ SELECT APPROX_COUNT_DISTINCT_DS_THETA(user_id_sketch,65536)...
○ APPROX_COUNT_DISTINCT_DS_THETA(expr, [size])

53. @ItaiYaffe, @yakiro
●
●
●
○

54. @ItaiYaffe, @yakiro
●
○
○
○
●
○
○
○
○

55. @ItaiYaffe, @yakiro
●
○
■
○
■
○

56. @ItaiYaffe, @yakiro
…

57. @ItaiYaffe, @yakiro
●
○
■
○
○
●
●
○

58. Time for questions
@ItaiYaffe
@yakiro
58
Thank you!
Apache Druid is an independent project of The Apache Software Foundation. More information can be found at https://druid.apache.org.
Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

59. Dates: TBD
imply.io
59
Register Now for
the Next Druid
Virtual Summit