Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NATS Streaming - an alternative to Apache Kafka?

1,959 views

Published on

A quick intro into message brokers and NATS Streaming compared to Apache Kafka

Published in: Software
  • Login to see the comments

NATS Streaming - an alternative to Apache Kafka?

  1. 1. NATS Streaming - an alternative to Apache Kafka? Anton Zadorozhniy
  2. 2. Agenda • Designs of message brokers • NATS project • NATS Streaming concepts • Limitations compared to Kafka • Advantages • Demo
  3. 3. What’s a message broker and how it’s used • Data management tool to communicate between two or more applications: • Pub/Sub, Point-to-point, Request/Response, Streams • Common uses: • Application integration • Data ingestion • Stream processing / low latency analytics
  4. 4. What’s inside message broker: two designs Classic design • ActiveMQ, Qpid, RabbitMQ, NATS • Features: • Routing/filtering • Re-queing • TTL/deadletter Commit log design • Kafka, Pulsar, NATS Streaming • Features: • Stream processing • Multiple consumers for the same stream • Persistence*
  5. 5. Classic design • Think of ArrayList • Broker tracks progress of each message, destroys message if it was consumed • Broker can re-que message if consumer failed to acknowledge consumption • Broker could implement TTL and other routing/filtering logic
  6. 6. Commit log design • Think of LinkedList • Only add messages to the head of list / log • Broker always stores messages, even if it was consumed multiple times (provides “time machine” capability) • Multiple consumers are cheap (no data duplication)
  7. 7. NATS project • Message broker written in Go with goals to be simple and fast • Maintained by Cloud Native Compute Foundation • Use cases: mobile apps, microservices and cloud native, IoT • https://nats.io • Initially in-memory message broker • Seriously fast, on good machine provides 20 mln msg/s • NATS Streaming is an extension for stream processing • Also checkout Liftbridge
  8. 8. NATS performance (a bit old)
  9. 9. NATS Streaming: concepts • Channels: persistent queues, similar to Kafka topics • Each channel is a ring buffer • Messages are written to channels even if there’s no subscription • Subscriptions: progress entity, similar to Kafka consumer groups • Messages are written to channels even if there’re no subscriptions • Sequence: position in ring buffer, similar to Kafka offset • Also allows positioning on timestamp
  10. 10. NATS Streaming: system architecture • Clustering: raft clusters with replicated state, only one machine is active • Fault tolerance: ability to use shared state between multiple NATS Streaming services • Partitioning: ability to assign different channels (“distribute channels”) across multiple NATS Streaming nodes to distribute load • Clustering and Partitioning/FT are incompatible!
  11. 11. Limitations comparing to Kafka • Replication is only available in clustering mode, with no scalability • No scalability for single channel • Instead of log controlled by retention time, NATS is using ring buffer – provides very little control on how data is deleted • Kafka persistence is actually really great (low latency), esp for large messages • Rather small ecosystem with little integrations available: • Nothing similar to Kafka Connect • No support from things like Flink or Spark Streaming
  12. 12. Advantages of NATS Streaming • Simplicity • Small footprint (12 MB statically linked binary, 20 MB for NATS Streaming) • Ability to use existing NATS clusters
  13. 13. Demo

×