At GameAnalytics we receive and process real time behavioural data from more than 100 million daily active users, helping thousands of game studios and developers understand user behaviour and improve their games. In this talk, you will learn how we managed to migrate our legacy backend system from using an in-house built streaming analytics service to Apache Druid, and the lessons learned along the way. By adopting Druid, we have been able to reduce development costs, increase reliability of our systems and implement new features that would have not been possible with our old stack. We will provide an overview of our approach to schema design, segments optimization, creation of our query layer, caching and datasources optimisation, which can help you better understand how you can successfully use Druid as a key component on your data processing and reporting infrastructure.

1. Building a Real-Time
analytics service with
Apache Druid
Virtual Druid Summit
October 2020
Ramón Lastres Guerrero, Director of Engineering,
GameAnalytics
1

2. Agenda
➢
➢
➢
➢
○
○
○
○
○
○
➢

3. Introduction to GameAnalytics
user behaviour analytics
focused on just gaming
SDKs
Rest API https://gameanalytics.com/docs/item/rest-api-doc
results in real-time and also historical aggregate

4. Introduction to GameAnalytics
150M+ 25,000+ 19B+ JSON1.7B+ 1 TB

5. 25,000 Daily Active Games

6. Analytics for 90,000 Game Developers

7. Key Performance Indicators: Player Retention

8. Interactive Filtering

9. Interactive Filtering

10. Technical Requirements
high level technical requirements
● (responsive Frontend)
● real time queries
● Reliability
● infrastructure cost
● flexible querying / filtering
● number of unique users

11. Backend Overview
three main components
●
●
●

12. Data Collection

13. Data Annotation System

14. Aggregation and Reporting: Druid
s3

15. Druid: Batch Ingestion Coordination

16. Druid: Query Layer
build our own query layer
● define metrics on backend side
● Implement authentication
● caching, query priorities, rate limiting
Elixir language
Druid client for Elixir

17. Druid: Imply Pivot
Imply Pivot

18. A / B testing and Druid
What is
A / B testing?

19. A / B testing and Druid
●
●
●

20. A / B testing and Druid

21. A / B testing and Druid
real-time
result metrics in real time
probabilistic model
variants are just Druid dimensions

22. Druid: Cluster Topology
Imply Cloud
●
●
●

23. Druid: Performance numbers
multi-tenancy
75k queries per hour
rollup
DAU 1.4k

24. Druid: Performance numbers

25. Hash partitioning VS single dimension partitioning
game_id (our tenant id) dimension
unstable EMR ingestion
hashed partitioning

26. Druid: Query Layer Caching
always implement good caching

27. Annotation System
annotation service

28. Annotation System
SDK attribution
partners

29. Annotation System: Calculating Player Retention
retention
calculation
increases the size by ~30%
installation timestamp (truncated to day)

30. Single Datasource VS Multiple Datasources
one single Kinesis stream
high cardinality low cardinality
reduce number of rows processed

31. Single Datasource VS Multiple Datasources
daily
Datasource Daily Size Daily segments Avg. Segment size
Small ~ 8.5GB ~ 10 ~ 550MB
Reduced ~ 50GB ~ 75 ~ 550MB
Full ~ 290GB ~ 450 ~ 550MB

32. Single Datasource VS Multiple Datasources
●
●
●
●
●
●

33. Single Datasource VS Multiple Datasources

34. Single Datasource VS Multiple Datasources

35. Druid: Tiering
leverage tiering
use it to lower costs
serving more frequently accessed data
with more powerful hardware

36. Druid: Lookups
joins with data stored outside of Druid
using lookups we can query on studio and
organization level

37. Time for questions
@gameanalytics
37
Thank you!
Apache Druid is an independent project of The Apache Software Foundation. More information can be found at https://druid.apache.org.
Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

38. Dates: November 10, 2020
druidsummit.org
38
Register Now for
the Next Druid
Virtual Summit