Slides posted with permission from eBay's Monitoring Platform team. The original talk was given at the Druid meetup on 2018-03-13, and the abstract was: "At eBay, we are using Druid for application monitoring and analytics in near real time. We process data coming from thousands of apps which generates more than 100’s of Billion events per day. Druid helped us to monitor critical events and health of our apps in near real time and scale for our needs. We will be presenting how we built and scale Druid at eBay for our use-case in detail. We will be sharing challenges, learnings, and our journey building on this platform at eBay."

2. Motivation
➢ Existing system is legacy product.
➢ Scaling challenges with architecture evolution.
➢ Cardinality explosion with Microservices.
➢ Missing features like percentiles, topN.

3. Data received from
Apps
Messaging Layer
Druid Deployment
Apps
Event
Publisher
Upstream
Processor
Kafka
Architecture

4. ➢ Built on Kubernetes
➢ Indexed at 1 min, 15 min, 1 hour. Reindex for 1 day
➢ Median Query response times < 1 second
➢ Smart granularity selection for different range queries
➢ HDFS for deep storage
Druid Deep Dive

5. Resiliency Architecture
➢ Dual-cluster
setup
➢ Smart query
routing
➢ Druid monitoring
through events
➢ Resilient infra
➢ Two replicas of
data per DC

6. Scale
5000+
Apps
4M+
Events/Sec
20M+
Unique
Dimension
Values
250+
Nodes in
Druid Cluster
100s B
Events
Processed/Day

7. ➢ Used by SREs to monitor application health
➢ Used by developer community to monitor their apps
➢ Anomaly detection
➢ Alerting
Customers & Usage

8. ➢ Percentiles
➢ Self-serve Alerting
Next steps