We went through the journey of deploying Apache Druid clusters on Kubernetes(K8s) and created a druid-operator (https://github.com/druid-io/druid-operator). This talk introduces the druid kubernetes operator, how to use it to deploy druid clusters and how it works under the hood. We will share how we use this operator to deploy Druid clusters at Splunk. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Druid is a complex stateful distributed system and a Druid cluster consists of multiple web services such as Broker, Historical, Coordinator, Overlord, MiddleManager etc each deployed with multiple replicas. Deploying a single web service on K8s requires creating few K8s resources via YAML files and it multiplies due to multiple services inside of a Druid cluster. Now doing it for multiple Druid clusters (dev, staging, production environments) makes it even more tedious and error prone. K8s enables creation of application (such as Druid) specific extension, called “Operator”, that combines kubernetes and application specific knowledge into a reusable K8s extension that makes deploying complex applications simple.

1. Druid K8S Operator
Himanshu Gupta
1

2. Agenda
◆ Few Druid Use Cases at Splunk
◆ Druid
◆ Kubernetes
◆ Kubernetes Operator
◆ Druid Kubernetes Operator

3. Splunk Business Flow: Process Mining

4. Splunk Observability: Trace Analysis

5. Druid: Extensibility
◆ Customizable Column Storage Format
◆ Customizable Aggregation
◆ Virtual Columns

6. Kubernetes Design Approach: Control
System
Target State, stored in
"API Server" backed
by etcd
Controller
Observed/Current
State
Actions

7. Kubernetes: Target State
Many type of Kubernetes Resources (typically YAMLs)
◆ StatefulSet
◆ Deployment
◆ Service
◆ Many Many more.....

8. Kubernetes: Deploying a WebService
◆ kubectl create -f my_StatefulSet.yaml
◆ kubectl create -f my_ConfigMap.yaml
◆ kubectl create -f my_Service.yaml

9. Kubernetes: Deploying a WebService
S
e
r
v
i
c
e
I
P
WebService-pod-0
WebService-pod-1
WebService-pod-2

10. Druid Architecture

11. Druid on K8S
So, You want to deploy Druid on K8S....

12. Druid on K8S

13. Druid on K8S
Router, Broker, Historical, Coordinator, Overlord, MiddleManager
Lots of K8S Resources...
◆ 6 StatefulSets
◆ 6 Services
◆ 6 + 1 ConfigMaps
◆ and more....

14. Druid on K8S: Operator Approach
Can we write another application that assumes domain
knowledge of both, Druid and K8S, which can be given a "Druid
Cluster Spec" and that application in turn talks to K8S and
creates/submits all the necessary YAML resource required to
manifest the "Druid Cluster".
We can deploy that application on K8S itself.

15. Druid on K8S: Operator Approach
◆ General and Easy to Reuse/Share
◆ Community Adoption and Hardening
◆ Manage "Lifecycle" not only "Deployment" of cluster
◆ Coded in a general purpose programming language

16. Kubernetes Operator
An application, deployed on K8S, that deploys/manages another
[more complex] application on K8S.

17. Kubernetes Operator
◆ Custom Resource
◆ Resource WATCH Api (notify me about xx resources)

18. Kubernetes Operator
Custom Resource,
stored in "API Server"
backed by etcd
Operator
Observed/Current
State
Actions

19. Druid K8S Operator
A K8S operator for managing lifecycle of Druid clusters.

20. Druid K8S Operator: Present
◆ Deploy
◆ Constant Reconciliation
◆ [Rolling] Update(version update, config changes, cluster topology changes)
◆ Destroy

21. Druid K8S Operator : Future
◆ Build improvements to incorporate Integration Testing
◆ Improved MiddleManager Upgrade Handling
◆ MiddleManager Autoscaling
◆ Druid Cluster Health Monitoring and Alerts
◆ Druid Overlord/Coordinator Dynamic Config, Coordinator
Rules management
◆ Druid Tasks Management
◆ Full-on cluster Autoscaling ?

22. 22

23. Druid K8S Operator : Getting Started
getting_started.md

24. Druid K8S Operator : Sample Tiny Druid
Cluster
getting_started.md

25. Druid K8S Operator : Sample Tiny Druid
Cluster

26. Druid K8S Operator : Debugging
getting_started.md

27. Druid K8S Operator: Debugging
kubectl logs druid-operator-85d6748dd5-gx9k6

28. Druid K8S Operator: Debugging
kubectl describe druid tiny-cluster

29. Druid K8S Operator: Debugging
kubectl describe druid tiny-cluster

30. Known Issues

31. Known Issues: PVCs not Deleted

32. Known Issues: Rolling Deployment gets Stuck

33. Known Issues: Further Manual Intervention
for example you updated something , which resulted in updation
of StatefulSet, Service i.e. a resource/field that can not be
updated, then operator would be stuck.

34. Known Issues: Further Manual Intervention

35. Druid K8S Operator: Source Code
OperatorSDK
Golang
https://github.com/druid-io/druid-operator/
druid_types.go ( Druid Cluster Spec)
handler.go:deployDruidCluster(..) (Main Control Loop)

36. Time for questions
36
Thank you!
Apache Druid is an independent project of The Apache Software Foundation. More information can be found at https://druid.apache.org.
Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

37. Dates: November 10, 2020
druidsummit.org
37
Register Now for
the Next Druid
Virtual Summit