Performance improvements done to RabbitMQ from releases 3.5.4, 3.5.5, 3.5.6 and 3.6.0

1. Improvements in
RabbitMQ Performance
and Future Directions
Alvaro Videla - RabbitMQ

2. @old_sound
github.com/videlalvaro

3. About Me
Co-authored
RabbitMQ in Action
http://bit.ly/rabbitmq

4. What is RabbitMQ

5. RabbitMQ

6. RabbitMQ

7. RabbitMQ
• Multi Protocol Messaging Server

8. • Multi Protocol Messaging Server
• Open Source (MPL)
RabbitMQ

9. • Multi Protocol Messaging Server
• Open Source (MPL)
• Polyglot
RabbitMQ

10. • Multi Protocol Messaging Server
• Open Source (MPL)
• Polyglot
• Written in Erlang/OTP
RabbitMQ

11. Multi Protocol
http://bit.ly/rmq-protocols

12. http://www.rabbitmq.com/community-
plugins.html
Community Plugins

13. Polyglot

14. Polyglot
• Java
• node.js
• Erlang
• PHP
• Ruby
• .Net
• Haskell

15. Polyglot
Even COBOL!!!11

16. Some users of RabbitMQ

17. • Instagram
• Indeed.com
• Telefonica
• Mercado Libre
• NHS
• Mozilla
Some users of RabbitMQ

18. The NewYork Times on
This architecture - Fabrik - has dozens of RabbitMQ instan
spread across 6 AWS zones in Oregon and Dublin.
Upon launch today, the system autoscaled to ~500,000 use
Connection times remained flat at ~200ms.
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-
January/032943.html

19. http://www.rabbitmq.com/
download.html
Unix - Mac - Windows

20. Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/
RabbitMQSimulator

21. http://tryrabbitmq.com

22. RabbitMQ Simulator

23. Performance
Improvements

24. The Problem

25. The Problem
./runjava.sh com.rabbitmq.examples.PerfTest
-e exchange_name -t topic -u queue_name
-f persistent -r 5000 -s 1000 -x1 -y0

26. The Problem
Publish 5000 1kb msgs/sec without consumers
Be able to sustain publish rate during one hour

27. The Problem
Big throughput drop around 1 minute
into the benchmark

28. The Problem
As soon as
Queue Paging kicked in
Performance Dropped

29. Culprits
• queue_index flush journal
• messages published into the queue index when
paging
• memory leaks related to refc binaries

30. Flushing the QI Journal
• Sparse Array by Message SeqIds
• Publishes, Delivers and Acks
• Queue Index is written in Append Only Segments
• RabbitMQ keeps an in memory Journal of ops to
prevent going to disk all the time

31. Flushing the QI Journal
queue_index_max_journal_entries = 65536

32. Flushing the QI Journal
entry_to_segment/3 called 65536 times

33. Flushing the QI Journal
entry_to_segment/3 called 65536 times
Appending data to a binary
and then writing that to disk

34. Solution
• Keep a Sparse Array as a cache of “entry to
segments”
• Flush said array to disk whenever the Journal
needs to be flushed

35. Solution
array:to_list(EntriesToSegment).

36. Solution
array:to_list(EntriesToSegment).
What happens with blank elements?

37. Solution
array:new([{default, []},
fixed,
{size, ?SEGMENT_ENTRY_COUNT}]).

38. Messages Publishes into the
Queue Index

39. queue_index_embed_msgs_below = 4096
Messages Publishes into the
Queue Index

40. Messages Publishes into the
Queue Index
• Implemented Batch Publishing into the Queue
Index
• Batch Handling of Delivers and Acks
• Flush Publishes, Delivers and Acks cache every
20.000 messages or when paging finished.

41. Queue Paging
Alphas -> Betas -> Deltas

42. binary memory leak
• Related to Paging
• discovered with recon
• forced queue process garbage collection after
paging
http://ferd.github.io/recon/index.html

43. The Problem II

44. The Problem II
Mirror Queue Synchronisation
“is too slow”

45. The Problem II
Initial tests showed that it took
60 seconds to sync 1MM messages

46. The Problem II
Original implementation was synching one
message at a time

47. Solution II
Why not implement batch
publishing at the queue level?

48. Batch Publishing
• Implemented for Backing Queue Implementations
• rabbit_variable_queue
• rabbit_priority_queue
• rabbit_mirror_queue_master
• rabbit_mirror_queue_slave

49. Solution II
Initial tests showed that it took
6 seconds to sync 1MM messages

50. The Problem III
Queues that support offline consumers for
24 hours or more

51. Solution III
• Lazy-Queue concept
• Disable in-memory message cache
• Only load messages in RAM if consumers are
present
• Implement Lazy Queues for all Backing Queue
implementations

52. Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
Initial tests show stable throughput comparable to “default”
queues with persistent messages

53. Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
10MM of 1KB use 1.5MB of RAM
vs
1.2GB of RAM for default queues

54. Lazy Queues
https://github.com/rabbitmq/rabbitmq-server/issues/351
40MB of RAM when a
consumer is active

55. Other Improvements
• moved to erlang.mk
• moving to Ranch/Cowboy and related libraries
• improved management UI performance
• improved rabbitmqctl performance

56. Other Improvements
• pluggable hashing algorithm for passwords
• queue master location strategies

57. Future
• add Raft based queue mirroring
• add Raft based cluster formation
• Re-thing rabbitmqctl to allow for extensions
• Tools to recover data from disk

58. Thanks
Alvaro Videla - RabbitMQ