At Stitch Fix, we maintain a distributed Kafka Connect cluster running several hundred connectors. Over the years, we've learned invaluable lessons for keeping our connectors going 24/7. As many conference goers probably know, event driven applications require a new way of thinking. With this new paradigm comes unique operational considerations, which I will delve into. Specifically, this talk will be an overview of: 1) Our deployment model and use case (we have a large distributed Kafka Connect cluster that powers a self-service data integration platform tailored to the needs of our Data Scientists). 2) Our favorite operational tools that we have built for making things run smoothly (the jobs, alerts and dashboards we find most useful. A quick run down of the admin service we wrote that sits on top of Kafka Connect). 3) Our approach to end-to-end integrity monitoring (our tracer bullet system that we built to constantly monitor all our sources and sinks). 4) Lessons learned from production issues and painful migrations (why, oh why did we not use schemas from the beginning?? Pausing connectors doesn't do what you think it does... rebalancing is tricky... jar hell problems are a thing of the past, upgrade and use plugin.path!). 5) Future areas of improvement. The target audience member is an engineer who is curious about Kafka Connect or currently maintains a small to medium sized Kafka Connect cluster. They should walk away from the talk with increased confidence in using and maintaining a large Kafka Connect cluster, and should be armed with the hard won experiences of our team. For the most part, we've been very happy with our Kafka Connect powered data integration platform, and we'd love to share our lessons learned with the community in order to drive adoption.

1. Kafka Connect
Operational Lessons Learned
From the Trenches
Liz Bennett
Software Engineer, Confluent

2. About Me
Liz Bennett
Currently @
As of last week! 😄
Formerly @
Formerly @
Formerly @
2

3. Get ready for a potpourri of Kafka Connect
operational advice, lessons learned, and
random information!
3

4. We’ll cover...
● Kafka Connect deployment and use cases
● Useful operational tools
● End-to-End monitoring
● Lessons learned from outages
4

5. Deployment Model and Use Cases

6. The Cluster
● Distributed Kafka Connect Cluster
● Self-service for Data Scientists
● ~50 workers
● ~500 connectors
● ~1000 tasks
● Low/medium Kafka volume
● 6 different connectors installed
○ RabbitMQ Source & Sink, Elasticsearch Sink, S3 Sink, SQS Source, Feature Store Sink
● All data is json formatted
6

7. 7

8. Read my blog post to learn more!
Putting the Power of Apache Kafka into the Hands of Data Scientists
8

9. Useful Operational Utilities
Jobs, Metrics, Alerts, Dashboards, Services, Transforms

10. Jobs
10

11. Jobs
● Restart crashed connectors 🎬
○ Post to slack, include crash reason
● Verify resources in external systems 🕵
● Health checks 😷
● Autoscale # of tasks ⚖
● Backup connect-configs topic 💾
○ Store in Github to easily review history
● Expire Elasticsearch indices ☠
● CI/CD jobs to build custom connectors 🚢
11

12. Dashboards &
Metrics
12

13. Dashboards & Metrics
● Kafka Connect heap usage dashboard and alerts
● Kafka consumer lags dashboard and alerts
● ❗Very Important❗: per topic messages/sec❗AND❗ bytes/sec
○ For debugging, alerting and auto-scaling
● # of tasks
● # of connectors
● # of open resources (e.g. TCP connections)
13

14. Single Message
Transforms
14

15. Single Message Transforms
Standard Transforms
● TimestampRouter
○ Used in Elasticsearch connectors
Custom Transforms
● RegexFieldFilter
○ Filter out messages by matching Regex to Json path
○ Skip corrupt data
● DropField
○ Delete fields, e.g. PII
○ Make Elasticsearch happy
15

16. Services
16

17. Kafka Connect
Admin Service
17

18. Admin Service
● Wraps Kafka Connect API 🎁
● Manages resources in external data systems 👉
○ Create tables in the Hive Metastore
○ Configure alerts
○ Create/dete topics
○ TODO: create index patterns in Kibana
● Sample topics in Kafka 🔬
18

19. Example: Kafka Connect S3 API
{
"connector.class": "io.confluent.connect.s3.S3SinkConnector", "s3.region": "us-east-1","topics.dir":
"dw","flush.size": "100","tasks.max": "1",
"timezone": "UTC","sf.s3.borderland.api.config": "foo.com”,"sf.metadata.owner.team": ":team:infra",
"transforms": "EventNameRegexFilter","sf.s3.hinterland.api.config": "foo.com,
"transforms.EventNameRegexFilter.field.path": "$.event_name","locale": "en","sf.s3.formatter.table.has.writets":
"true",
"sf.s3.partitioner.db.name": "db_name",
"format.class": "com.stitchfix.datahighway.sfs3connector.SFS3FormatDailyPartitions",
"schema.generator.class":
"io.confluent.connect.storage.hive.schema.TimeBasedSchemaGenerator","sf.s3.formatter.event.name.regex.match.list":
"^.*$",
"sf.s3.formatter.payload.field": "$.payload",
"sf.s3.formatter.field.timestamp.transformers.map": "{"$.metadata.timestamp":"yyyy-MM-dd
HH:mm:ss.SSSSSS","$.payload.ts_posted":"yyyy-MM-dd HH:mm:ss.SSSSSS"}",
"s3.bucket.name": "stitchfix.aa.event","partition.duration.ms": "3600000","sf.s3.formatter.timestamp.field":
"$.metadata.timestamp",
"topics": "my_topic","sf.s3.partitioner.table.name": "my_table_in_s3","sf.s3.formatter.s3.timestamps.format":
"yyyy-MM-dd HH:mm:ss.SSS000",
"transforms.EventNameRegexFilter.regex": "^.*$","partitioner.class":
"com.stitchfix.datahighway.sfs3connector.SFS3PartitionerDailyPartitions",
"name": "my-s3-connector","sf.metadata.owner.email": "foo.bar@stitchfix.com",
"storage.class": "io.confluent.connect.s3.storage.S3Storage","rotate.schedule.interval.ms": "30000",
"path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH","transforms.EventNameRegexFilter.type":
"com.stitchfix.datahighway.transforms.FieldRegexFilter",
"schemas.enable": "false","sf.s3.formatter.event.name.field": "$.event_name"
}
19

20. Example: Wrapped API
{
"processing_threads": 1,
"name": "my-s3-connector",
"topics": ["my_topic"],
"table_name": "my_table_in_s3",
"flush_size": 100,
"flush_interval_seconds": 30,
"owner_email": "bob.ross@happytrees.com",
"owner_team": ":team:painters"
} 20

21. API Wrapper
● Hard codes default values
○ AWS defaults
○ Schema related defaults
○ Security related defaults
○ Kafka hostnames
○ Kafka Connect class name configurations
● Injects custom fields
○ Connector owner metadata
21

22. API Wrapper Design
Requirements
● Adding new connectors should be very easy
● Adding new fields to connectors should very easy
● Typos should be hard to make, ideally statically enforced by compiler
Design
● Redesigned 3 times
● Used Jackson Json Annotations for Json SerDe
● Very hierarchical OOP design
● Lots of tests...
22

23. API Wrapper Lessons Learned
Pros 👍
● Greatly simplified the web UI React code
● Exposed simple API for other teams to script against
● Consolidated default configs in one place
● Totally abstracts Kafka Connect
Cons 👎
● Friction when installing new connectors
23

24. End-to-End Monitoring

25. End-to-End Monitoring Requirements
● Monitor for
○ Delayed data
○ Duplicated data
○ Previously delivered data has disappeared
○ “Late arriving” data is not correctly delivered
○ Failures to publish data
● All ingresses and egresses are monitored 24/7
● Use user-level interfaces only
● ❗Very important❗: staging must have all the same monitoring as prod!!
25

26. Tracer Bullets
26

27. Tracer Bullets Overview
● Send synthetic events (“tracer bullets”) to all ingress points
● Funnel all tracer bullets into tracer_bullet topic
● Create at least 1 of every kind of Sink Connector consuming from the
tracer_bullet topic
● Create tracer bullets with logical timestamps
● Send current and “late arriving” data
● Persist metadata for tracer bullets in external system (i.e. Elasticsearch)
27

28. 28

29. 29

30. 30

31. 31

32. 32

33. Tracer Bullet Payloads
Must include at least
● Unique id
● ❗Very important❗ LOGICAL timestamp (a.k.a event time)
33

34. Tracer Bullets Metadata
Tracer bullet metadata should include
● Timestamp sent
● Tracer bullet’s unique ID
● Ingress point
● Expected egress points
● Send status (SUCCESS, SEND_FAILED, etc.)
● # of duplicates found in sink
● Delivery timestamp (if delivered)
● Environment (dev/staging/prod)
34

35. Tracer Bullet Polling
● Continuously queries every destination
○ Queries a sliding window of several hours
○ Queries the “late arriving” tracer bullets
● Check for duplicates
● Updates metadata when tracer bullets are delivered (or disappear!)
35

36. Tracer Bullet Alerting
● Poll the tracer bullet metadata and alert when:
○ Duplicates were found
○ Tracer bullets are delayed (according to your SLAs)
○ Previously delivered tracer bullets mysteriously disappear
○ Tracer bullets failed to send in the first place
● Alert if there is no tracer bullet metadata
36

37. Tracer Bullet Dashboards
37

38. Tracer Bullet Summary Job ❗Very Important❗
● Generate daily reports
○ Overall end-to-end latency
○ No. of tracer bullets sent per sink/source
● Indicates tracer bullets are functioning properly
38

39. Tracer Bullets Lessons Learned
● Make them extensible!
● Tracer bullets: just a big streaming application… write good tests!
● Dev environments are hard…
● Tracer bullets allowed for fewer integration tests
39

40. Lessons Learned the Hard Way
Production Issues, Annoying Gotchas

41. Pausing Connectors
● PUT /connectors/my-connector/pause
● What does pausing connectors actually do?
○ Pauses poll loop
● How to really stop a connector?
○ Delete and recreate
○ Shut down Kafka Connect
41

42. Jar Hell
[2018-02-18 10:28:33,290] INFO Loading plugin from: /home/ubuntu/kafka_2.11-1.0.0/plugins/lib
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:179)
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
at org.reflections.Reflections.<init>(Reflections.java:126)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)f
42

43. Jar Hell
[2018-02-18 10:28:33,290] INFO Loading plugin from: /home/ubuntu/kafka_2.11-1.0.0/plugins/lib
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:179)
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
at org.reflections.Reflections.<init>(Reflections.java:126)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
at
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)f
43

44. Jar Hell
● As of version 0.11.0.0 (CP 3.3.0), Kafka Connect has plugin.path
● USE PLUGIN.PATH!!!!
44

45. Schemas
● Migrating to schemas is really tough
● Use schemas from the beginning
● Some connectors require schemas
○ JDBC Sink Connector
45

46. Rebalancing
What is it
● Created/deleted connectors & added/removed workers trigger rebalances
Why is it bad
● API becomes unresponsive (responds with 409 status)
● Thundering rebalances caused instability in Elasticsearch
● S3 Connector + RegexFieldFilter transform caused big problems
● As of 2.3.0 (CP 5.3.0) rebalancing is much better!!! Upgrade, upgrade upgrade!
○ Shout out to Konstantine Karantasis for Cooperative Incremental Rebalances
46

47. Conclusion

48. Conclusion
● Kafka Connect is awesome! 😄😄😄
● Metrics 📈
○ ❗Very Important❗: per topic message/sec❗AND❗ bytes/sec
● Write an admin service to tie together your whole system 🎁
● End-to-end monitoring 🔥🔥🔥
○ ❗Very Important❗: have monitoring in both prod and staging
○ ❗Very Important❗: generate daily summary reports
○ ❗Very Important❗: universal support for logical timestamps
● Upgrade early and upgrade often! 💯
48

49. Questions?
49

50. Thank you!
lbennett@confluent.io ✉
@zzbennett