Change data capture with MongoDB and Kafka.
โข
12 likesโข7,726 views
In any modern web platform you end up with a need to store different views of your data in many different datastores. I will cover how we have coped with doing this in a reliable way at State.com across a range of different languages, tools and datastores.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Change data capture with MongoDB and Kafka.
Similar to Change data capture with MongoDB and Kafka. (20)
More from Dan Harvey
Recently uploaded
Recently uploaded (20)
Change data capture with MongoDB and Kafka.
- 1. Change Data Capture with Mongo + Kafka By Dan Harvey
- 3. High level stack React.js - Website Node.js - API Routing Ruby on Rails + MongoDB - Core API Java - Opinion Streams, Search, Suggestions Redshift - SQL Analytics
- 4. Problems โข Keep user experience consistent โข Streams / search index need to update โข Keep developers ef๏ฌcient โข Loosely couple services โข Trust denormalisations
- 5. Use case โข User to User recommender โข Suggest โinterestingโ users to a user โข Update as soon as you make a new opinion โข Instant feedback for contributing content
- 6. Log transformation Java$Services Avro Rails$API JSON/BSON Mongo Opinion Optaileroplog Kafka: User Topic User Recommender Change$data$capture Stream$processing User Kafka: Opinion Topic
- 7. Op(log)tailer โข Converts BSON/JSON to Avro โข Guarantees latest document in topic (eventually) โข Does not guarantee all changes โข Compacting Kafka topic (only keeps latest)
- 8. Avro Schemas โข Each Kafka topic has a schema โข Schemas evolve over time โข Readers and Writers will have different schemas โข Allows us to update services independently
- 9. Schema Changes โข Schema to ID managed by Con๏ฌuent registry โข Readers and writers discover schemas โข Avro deals with resolution to compiled schema โข Must be forwards and backwards compatible Ka#a$message:$byte[] message:$byte[]schema$ID:$int
- 10. Search indexing โข User / Topic / Opinion search โข Re-use Kafka topics from before โข Index from Kafka to Elasticsearch โข Need to update quickly and reliably
- 11. Samza Indexers โข Index from Kafka to Elasticsearch โข Used Samza for transform and loading โข Far less code than Java Kafka consumers โข Stores offsets and state in Kafka
- 12. Elasticsearch Producer โข Samza consumers/producers deal with I/O โข Wrote new ElasticsearchSystemProducer โข Contributed back to Samza project โข Included in Samza 0.10.0 (released soon)
- 13. Samza Good/Bad โข Good API โข Simple transformations easy โข Simple ops: logging, metrics all built in โข Only depends on Kafka โข Inbuilt state management โข Joins tricky, need consistent partitioning โข Complex ๏ฌows are hard (Flink/Spark better)
- 14. Decoupling Good/Bad โข Easy to try out complex new services โข Easy to keep data stores in sync, low latency โข Started to duplicate core logic โข More overhead with more services โข Need high level framework for denormalisations โข Samza SQL being developed
- 15. Ruby Workers โข Ruby Kafka consumers not greatโฆ โข Optailer to AWS SQS (Shoryuken gem) โข No order guarantee like Kafka topics โข But guaranteed trigger off database writes โข Better for core data transformations
- 16. Future โข Segment.io user interaction logs to Kafka โข Use in product, view counts, etcโฆ โข Fill Redshift for analytics (currently batch) โข Kafka CopyCat instead of our Optailer โข Avro transformation in Samza
- 17. Questions? โข email: dan@state.com โข twitter: @danharvey