IBM MQ and Kafka, what is the difference?
•
3 likes•1,633 views
Message queueing solutions used to be the one general purpose tool used for all asynchronous application patterns, then along came event streaming as an application model. To support this effectively needed a whole new approach to how messages are handled by the messaging technology. Now the tables are turned and many are wondering if an event streaming solution can be used for all their asynchronous application patterns from now on. But just as message queueing solutions work in a way to optimize for their core use cases, so do event streaming solutions, and these behaviors directly affect the applications that use them. This session picks IBM MQ and Kafka to look at how they compare and, more importantly, differ in their behavior so that you can decide which application scenarios are best suited by each. Spoiler -they're both good in their own way!
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to IBM MQ and Kafka, what is the difference?
Similar to IBM MQ and Kafka, what is the difference? (20)
More from David Ware
More from David Ware (7)
Recently uploaded
Recently uploaded (20)
IBM MQ and Kafka, what is the difference?
- 1. TechCon 2020 IBM MQ and Kafka, what is the difference? David Ware IBM MQ Chief Architect
- 2. Messaging is essential for building fully connected, efficient and scalable solutions. More now than ever before Messaging covers a broad spectrum of requirements
- 3. Messaging Scalability Security Management High Availability There are many capabilities essential to every enterprise messaging solution IBM MQ and Kafka have both repeatedly been proven to tick these boxes…
- 4. © 2020 IBM Corporation So just pick one and move on?
- 5. © 2020 IBM Corporation So just pick one and move on? Perhaps, if you think everything looks like a nail
- 6. © 2020 IBM Corporation So just pick one and move on? Perhaps, if you think everything looks like a nail Otherwise, let’s focus on what you’re trying to achieve…
- 7. Critical data exchange: work that needs to be done Critical applications demand assured asynchronous interactions Essential capabilities: End-to-end once and once only Deliveryü Fine grain messaging Messages typically represent commands, queries and operations The message is a way to pass control from the originator of the message to the consumer
- 8. Event Driven: building scalable microservices Microservices increases the need for communication. API-based interactions can build fragile and unscalable tight bonds between components Topics and Subscriptions Publishing and subscribing to events relaxes the coupling of microservices Events are messages that communicate that something has occurred Essential capabilities:
- 9. Event Streaming: the expanding need for messaging Event Streaming brings data together from disparate sources, enabling even more responsive and engaging experiences for a wider set of users Scalable Subscription Stream History Efficient cloud and analytics applications utilize local decoupled buffers of event data Essential capabilities:
- 10. The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
- 11. © 2020 IBM Corporation To choose the right tool you’re probably going to need to go to the next level…
- 12. © 2020 IBM Corporation Message consumption
- 13. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption
- 14. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
- 15. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7
- 16. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
- 17. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
- 18. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 … • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
- 19. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
- 20. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
- 21. © 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER 0 1 23 45 67
- 22. © 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots
- 23. © 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots Kafka topic partitions…
- 24. © 2020 IBM Corporation Queues and topics
- 25. © 2019 IBM Corporation Comparing queues and topics QUEUE 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER PRODUCER PRODUCER 0 1 23 45 6 7 TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER CONSUMER 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Queues Topics • Each messages goes to a single consumer from the queue • Each message goes to every subscriber to the topic
- 26. © 2019 IBM Corporation CONSUMERCONSUMER MQ and Kafka topics TOPIC PARTITION 0 1 2 3 4 5 6 7 CONSUMER GROUP CONSUMER GROUP PRODUCER PRODUCER CONSUMER CONSUMER TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER 1 2 3 4 5 6 70 1 2 3 4 5 6 70 IBM MQ Kafka
- 27. © 2019 IBM Corporation Queues and topics IBM MQ Kafka Queues Primary MQ concept Queues are administratively or dynamically created Queues can be partitioned for scale All messages are stored on queues, even publications n/a Topics Core MQ capability Topics are dynamically created as needed by applications Each subscribing application has access to a dedicated queue of to-be-consumed messages matching their subscription. Primary Kafka concept Rolling history of messages maintained by Kafka. Topics can be partitioned for scale… Consumer’s offset within a shared stream of messages is maintained by Kafka.
- 28. © 2020 IBM Corporation Scaling
- 29. Horizontal scaling – workload balance the messages PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER IBM MQ Kafka PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER
- 30. Kafka Clusters • A Kafka system is typically made up of multiple active nodes • Topics can be partitioned to enable workload balancing across those nodes, improving scale and availability of the topics • Kafka workload balances messages across partitions, either randomly or based on application provided context to enable per-partition ordering • Each topic partition is active in one broker at a time • The messages held in a topic partition can be replicated to other brokers to ensure data integrity across failures and automatic recovery
- 31. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 Kafka partitions and consumer groups 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
- 32. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
- 33. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
- 34. © 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups - p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers X
- 35. IBM MQ Clusters and workload balancing PRODUCE MESSAGES TO QUEUE A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER • A single running instance of MQ is a queue manager. • These can act independently or collectively to build horizontally scaled solutions. • An MQ Cluster is a dynamically provisioned set of queue managers with the ability to partition queues across them. • MQ will workload balance inbound messages from producing applications across each partition of the queue. • With MQ’s Uniform Cluster capability, MQ will workload balance multiple consuming applications across each queue manager, ensuring messages are consumed efficiently. • MQ does not provide simple message ordering in these topologies • MQ’s HA technologies provide data integrity across failures and automatic recovery
- 36. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances
- 37. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20, … 01,11, … X
- 38. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20 01,11 03,13,23
- 39. © 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20,40 01,11 03,13,23,43 CONSUMER CONSUMER CONSUMER CONSUMER 30,50,70 32,52 31 21
- 40. © 2019 IBM Corporation Scaling IBM MQ Kafka Consuming application is the bottleneck Scale out the consuming application Messaging system is the bottleneck Scale out the messaging system Increase the number of consuming applications from the existing queue/subscription. Increase the number of topic partitions to support a greater number of consuming applications in a consumer group. Increase the number of queue managers deployed and workload balance messages and consumers across them. Increase the number of Kafka broker nodes deployed and redistribute the topic partitions across them. †Good practice is to capacity plan for highest load and pre-define multiple topic partitions to support more consumers than you currently require, enabling more dynamic consumer scaling https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ † † † Unlike MQ, where new queue managers do not require copies of the existing messages, Kafka requires event histories to be copied to a new broker before it can be effective, slowing down dynamic scaling https://www.infoq.com/presentations/cloud-native-kafka-netflix/ ††
- 41. © 2020 IBM Corporation Conversational messaging
- 42. © 2019 IBM Corporation ConversationConversation ConversationConversation Fine grain communication? • MQ is designed for tens of thousands of lightweight queues and hundreds of thousands of dynamic topic strings per queue manager • This enables each conversation (e.g. reply flow) to be a dedicated queue or topic • Each consumer only sees its intended message • Dynamic create/deletion of queues and topic strings IBM MQ Equivalent Kafka model … • Kafka is designed for less, larger, configured topic partitions per broker • Recommendations are typically for 2-4k topic partitions per broker* • Consumers process every message in the topic from a point forward • This model is considered a Kafka anti-pattern. Recommendation is to rearchitect application to a stream processing pattern if possible** • https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions ** https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka PRODUCERPRODUCER PRODUCERPRODUCER CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … TOPIC 2 30 1 … …
- 43. © 2020 IBM Corporation Message delivery
- 44. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Any problem in the messaging system or the application may result in messages being lost Messages are not lost in the event of a failure, but any message may be duplicated Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication An application can coordinate the processing of a message with a specific update to an external resource, such as a database
- 45. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications
- 46. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications Ideal for event streaming use case Ideal for critical data exchange use case
- 47. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Higher levels of message reliability typically reduce the need for applications to consider error scenarios. However, higher message reliability typically reduces overall message performance and can limit overall availability The trade off Event streaming criticaldata exchange Reliability Performance
- 48. © 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Kafka IBM MQ https://medium.com/@andrew_schofield/does-apache-kafka-do-acid-transactions-647b207f3d0e Sweet spots Default behavior. Some messages may be lost following an application or Kafka failure, but others persisted Default behavior. All messages of this type lost following an MQ failure by design Kafka well designed for this, producers can be configured to retry failed sends and consumers typically periodically update their offset Possible to design an application to behave like this through message batching and protocol optimizations, but not a key MQ scenario Possible within a closed Kafka system, but much harder with external systems. Technically possible to lock Kafka down to confirm every message for individual acknowledgment but loses benefits A natural fit for the destructive consume approach of MQ coupled with its core persistence model Supports Kafka-only transactions for exactly-once stream processing applications, but no external resource coordination A core MQ capability is as a resource manager (or coordinator) for global transactions Event streaming criticaldata exchange
- 49. © 2020 IBM Corporation Message availability
- 50. © 2019 IBM Corporation Per-message availability IBM MQ Kafka Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes. PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9
- 51. © 2019 IBM Corporation Per-message availability IBM MQ Kafka PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 active/active partitioning of queues and topics, each partition is active/passive Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes.
- 52. © 2020 IBM Corporation Summary
- 53. © 2020 IBM Corporation If you want to build the best solution, you’re probably going to need more than a hammer
- 54. The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
- 55. Thank you David Ware IBM MQ Chief Architect — www.linkedin.com/in/dware1 © Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are trademarks of IBM Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available at Copyright and trademark information. 57TechCon 2020 © 2020 IBM Corporation
- 56. TechCon 2020 / © 2020 IBM Corporation 58