RabbitMQ is the most popular open-source message broker. It’s a de facto standard for message-based architectures. And yet, despite the abundant documentation and usage, developers and operators can still get tripped up on configuration and usage patterns. Let’s face it: some of these best practices are hard to capture in docs. There’s a subtle difference between what RabbitMQ *can* do, and *how* you should use it in different scenarios. Now is your chance to hear from seasoned RabbitMQ whisperers, Jerry Kuch and Wayne Lund. Join Pivotal’s Jerry, Senior Principal Software Engineer, and Wayne, Advisory Data Engineer, as they share their top ten RabbitMQ best practices. You’ll learn: - How and when—and when *not*—to cluster RabbitMQ - How to optimize resource consumption for better performance - When and how to persist messages - How to do performance testing - And much more! Speakers: Jerry Kuch, Senior Principal Software Engineer Wayne Lund, Advisory Data Engineer, Pivotal

1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved.
Jerry Kuch - jkuch@pivotal.io
Wayne Lund - wlund@pivotal.io
12 Dec 2018
10 Things Every
Developer Using RabbitMQ Should Know

2. Introduction
● RabbitMQ is a complex, multi-
faceted system
● RabbitMQ has many features
● A few basic ideas can go a
long way in:
○ Understanding
RabbitMQ
○ Using RabbitMQ
effectively
Broker
Producer
Message
Exchange
Bindings
Queues
Consumer

3. Agenda
●Ten Topics
●From Three Major Themes:
○ Guiding Principles
○ Messages and Their Handling
○ Resource Management
●Where to Go Next
Broker
Producer
Message
Exchange
Bindings
Queues
Consumer

4. Guiding Principles
Ten Best Practices

5. ● RabbitMQ is not a database
● Aim to keep queue lengths short in practice
○ Balancing ingress and egress rates is the
first, best idea
○ Know what a “normal” length for a given
queue is
● Problems with long queues:
○ Memory alarms, paging to disk, TCP back
pressure
● Mind message sizes
● Monitor!
Guiding Principles
Item 1: “A Happy Rabbit
is an Empty Rabbit"

6. ● RabbitMQ is written in Erlang
● Erlang concurrency is based on lightweight processes
○ Rabbit is built of a collection of these communicating
processes
○ Each queue you create has its own queue process
○ Erlang processes are scheduled on a pool of OS
threads
○ Queues are thus natural concurrency bottlenecks
● Mechanical sympathy for your system helps you better
exploit it
● By contrast exchanges are not backed by their own
processes
Guiding Principles
Item 2: “The Queue is
Rabbit’s
Basic Unit of
Concurrency"

7. Messages and
Their Handling
Ten Best Practices

8. ● What makes us think anything happened in a
distributed system?
● Transfer of Responsibility:
○ At any moment, know who is responsible for
the existence and integrity of a message
○ Maintain the responsibility and hand it off in
a disciplined way
● Consumer side: AMQP ACKs.
● Producer side: Publisher Confirms (also AMQP
transactions)
Messages and
Their Handling
Item 3: The “Transfer of
Responsibility” Notion

9. ● Ways to consume: basic.get and basic.consume
● Most clients should favor basic.consume
● Prefetch (prefetch_count in basic.qos method):
○ How many messages a client is willing to
take at a time
○ Broker will deliver up to that many and wait
for ACK
● Small prefetches can inhibit throughput
● Large prefetch can result in one of many
consumers on a queue monopolizing queue’s
output
● Choose based on how fast consumer gets work
done before ACKing
Messages and
Their Handling
Item 4: Consumers
Should Eat Not Nibble

10. ● Messages in many applications break into
natural tiers.
● Many RabbitMQ features impose non-trivial
costs:
○ Persistence
○ HA Mirroring of Queues
○ Clustering
○ Publisher Confirms and AMQP Transactions
○ Consumer ACKs vs noACK/autoACK
● Think about your traffic, the benefits of a feature
and spend accordingly!
Messages and
Their Handling
Item 5: Know Costs of
Guarantees and Don’t
Overpay Unnecessarily

11. Resource
Management
Ten Best Practices

12. ● Some objects are more heavyweight than others
● CPU and RAM are limitations more often than
networking
● Management can impose overhead:
● Beware in large clusters of ‘detailed’ metrics
● Don’t hit the wall (and understand the padding in
front of it)!
● Padding: Memory and disk alarms, TCP back
pressure
● The Wall: Fatal resource exhaustion is fatal
Resource
Management
Item 6: Don’t Overtax
Your
System’s Resources

13. ● Understand AMQP Connections vs. Channels
○ Connections are expensive
○ Channels try to amortize the expense via
multiplexing
● Don’t churn connections and mind your OS limits
● Don’t share channels between threads
● Consider separate connections for publishers
and consumers
Resource
Management
Item 7: Manage
Connections and
Channels Thoughtfully

14. ● Auto-delete queues: delete when last subscriber
unsubscribes.
● Exclusive queues: exclusive to one connection,
deleted when connection closes.
● Queue length limit: as number of messages or
count of bytes;
● Per-Queue Message TTL
● Per-Message TTL
● Queue TTL: whole queue vanishes after TTL if no
consumers
Resource
Management
Item 8: Tips for Cleaning
Up Queues

15. ● Consider virtual hosts—remember they’re only
namespaces.
● Beware noisy neighbors!
● Sometimes separate clusters can be best:
○ Keep noisy neighbors apart
○ Easier to debug, determine culpability, etc.
Resource
Management
Item 9: Think About
Isolation

16. ● Monitoring! Web UI, management HTTP API, logs
○ Workloads:
https://github.com/rabbitmq/workloadsBuilt-
in liveness check API
● Can wire API to Splunk, Nagios, etc.
● Monitor not just node health but also your
message fabric:
○ Its shape and typical parameters
● Management statistics collection: collection rate
configurable, beware heavy load
● Read, Learn, Experiment, Understand, Repeat.
Resource
Management
Item 10: Know What’s
Happening

20. Guiding Principles
● A Happy Rabbit is an Empty Rabbit
● The Queue is Rabbit’s Unit of Concurrency
Messages and Their Handling
● The “Transfer of Responsibility” Notion
● Consumers Should Eat, Not Nibble
● Know Costs of Guarantees and Don’t Overpay
Unnecessarily
SUMMARY

21. Resource Management
● Don’t Overtax Your System’s Resources
● Manage Connections and Channels Thoughtfully
● Tips for Cleaning Up Queues
● Think About Isolation
● Know What’s Happening
SUMMARY
(Continued)

22. ● RabbitMQ Website: http://www.rabbitmq.com
● RabbitMQ in Action, Manning Publications
● RabbitMQ in Depth, Manning Publications
● Workloads: https://github.com/rabbitmq/workloads
● Marcial Rosales developer guide:
https://github.com/MarcialRosales/rabbitmq-developer-guide
● Jack Vanlightly Blogs: https://jack-vanlightly.com/
● CloudAMQP Best Practices:
https://www.cloudamqp.com/blog/index.html
Where to Learn More

23. Transforming How The World Builds Software
© Copyright 2018 Pivotal Software, Inc. All rights Reserved.