Queues queues queues — How RabbitMQ enables reactive architectures
•
11 likes•3,299 views
How reactive microservices based software architecture is built at Pipedrive, and what are the lessons learned with using RabbitMQ since 2012.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Queues queues queues — How RabbitMQ enables reactive architectures
Similar to Queues queues queues — How RabbitMQ enables reactive architectures (20)
Recently uploaded
Recently uploaded (20)
Queues queues queues — How RabbitMQ enables reactive architectures
- 1. Martin Tajur CTO, Co-Founder February 16, 2016 DevClub XLIII meetup, Tallinn, Estonia Hello.
- 2. ● Web-based CRM for small teams with big ambitions.
- 3. ● Web-based CRM for small teams with big ambitions. ● Founded in 2010.
- 4. ● Web-based CRM for small teams with big ambitions. ● Founded in 2010. ● Used by over 30,000 businesses worldwide.
- 5. ● Web-based CRM for small teams with big ambitions. ● Founded in 2010. ● Used by over 30,000 businesses worldwide. ● 140+ employees, venture funded (BVP, Series A in 2015)
- 6. ● Web-based CRM for small teams with big ambitions. ● Founded in 2010. ● Used by over 30,000 businesses worldwide. ● 140+ employees, venture funded (BVP, Series A in 2015) ● Engineering, Product, UX, Marketing in Tallinn and Tartu. Marketing, BizDev in New York, NY.
- 7. ● Web-based CRM for small teams with big ambitions. ● Founded in 2010. ● Used by over 30,000 businesses worldwide. ● 140+ employees, venture funded (BVP, Series A in 2015) ● Engineering, Product, UX, Marketing in Tallinn and Tartu. Marketing, BizDev in New York, NY. ● Very end user focused, helping the actual sales person do their job.
- 9. ● 20,000+ simultaneous online users
- 10. ● 20,000+ simultaneous online users ● 800+ API req/sec
- 11. ● 20,000+ simultaneous online users ● 800+ API req/sec ● 400,000+ incoming emails per day
- 12. ● 20,000+ simultaneous online users ● 800+ API req/sec ● 400,000+ incoming emails per day ● Started with Node.js based microservices, reactive architecture in 2012
- 13. ● 20,000+ simultaneous online users ● 800+ API req/sec ● 400,000+ incoming emails per day ● Started with Node.js based microservices, reactive architecture in 2012 ● In production with first Docker based services
- 14. ● 20,000+ simultaneous online users ● 800+ API req/sec ● 400,000+ incoming emails per day ● Started with Node.js based microservices, reactive architecture in 2012 ● In production with first Docker based services ● In total, 500+ VMs/hosts/instances
- 18. Queues, queues, queues How RabbitMQ enables reactive architectures Martin Tajur CTO, Co-Founder February 16, 2016 DevClub XLIII meetup, Tallinn, Estonia
- 19. ➔ Co-Founder and CTO of Pipedrive. ➔ Started career in 2001 as a designer, later a full stack dev. ➔ Part of ex-Skype mafia About me
- 23. Manifestos are good when ➔ they are used as building blocks ➔ they add, rather than compete or subtract ➔ they tap into the growing wisdom, and extend it
- 26. An initiative to define a tech agnostic shared vocabulary of architectural patterns for reactive systems.
- 28. Systems should be reactive to
- 29. Systems should be reactive to Events
- 30. Systems should be reactive to Events Load
- 31. Systems should be reactive to Events Load Failure
- 32. Systems should be reactive to Events Load Failure Users
- 33. Systems should be reactive to tolerate Events Load Failure Users
- 34. Responsive React to users in timely manner http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385 Goal
- 35. Responsive React to users in timely manner Resilient React to failures Elastic React to load http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385 Goal Principles
- 36. Responsive React to users in timely manner Resilient React to failures Elastic React to load Message-driven Component-to-component interaction http://www.slideshare.net/RezaSamee/the-reactive-manifesto-49897385 Goal Principles Method
- 37. Systems built as Reactive Systems are more flexible, loosely-coupled and scalable.
- 38. This makes them easier to develop and amenable to change.
- 39. They are significantly more tolerant of failure and when failure does occur they meet it with elegance rather than disaster.
- 40. Reactive Systems are highly responsive, giving users effective interactive feedback.
- 41. How queues fit in to the picture?
- 42. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication http://www.iron.io/top-10-uses-for-message-queue/
- 43. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 44. Decoupling ➔ Hard to predict future needs at the start of a project. ➔ Message queues create an implicit, data-based interface that different services can implement. ➔ Allows you to extend and modify processes independently, ensuring they adhere to the same interfaces. ➔ Lets you swap, mix and add queue consuming services.
- 45. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 46. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 47. Redundancy ➔ Sometimes, services die when processing data. ➔ Unless that data is persisted, it’s lost forever. ➔ Queues mitigate this by persisting data until it has been fully processed. ➔ No job gets lost.
- 48. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 49. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 50. Load Balancing, Scalability ➔ It’s easy to scale up the rate with which messages are added to the queue or processed – by adding more instances. ➔ Easy to create load balancing by attaching multiple workers to a single queue. ➔ No code changes, no configurations need to be tweaked. ➔ Scaling is as simple as adding more power.
- 51. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 52. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 53. Elasticity & Spikability ➔ When your application hits the front page of Hacker News, you’re going to see unusual levels of traffic. ➔ Your application needs to be able to keep functioning. ➔ But the traffic is anomaly, not the standard — it’s wasteful to have enough resources on standby to handle these spikes. ➔ Message queues will allow components to struggle through the increased load, instead of getting overloaded with requests and failing completely. ➔ Queue lengths and consumer utilization = basis for auto-scaling.
- 54. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 55. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 56. Resiliency ➔ When part of your architecture fails, it doesn’t need to take the entire system down :-) ➔ Message queues decouple services, so if a service that is processing messages from the queue fails, messages can still be added to the queue to be processed when the system recovers. ➔ This ability to accept requests that will be retried or processed at a later date is often the difference between an inconvenienced customer and a frustrated customer.
- 57. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 58. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 59. Delivery guarantees ➔ The redundancy provided by message queues guarantees that a message will eventually be processed. ➔ No matter how many processes are pulling data from the queue, each message will be processed at least once. ➔ This is often made possible using a way to “reserve” messages being processed, temporarily removing them from the queue. ➔ Unless the client specifically states that it’s finished with that message, the message will be placed back to the top of the queue.
- 60. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 61. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 62. Ordering Guarantees ➔ In a lot of situations, the order with which data is processed is important. ➔ Message queues are inherently ordered, and capable of providing guarantees that data will be processed in a specific order. ➔ Most message queues use FIFO (first in, first out), so the order in which messages are placed on a queue is the order in which they’ll be retrieved from it. ➔ (Beware when using multiple consumers)
- 63. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 64. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 65. Buffering ➔ In any non-trivial system, there are going to be components that require different processing times. ➔ For example, it takes less time to upload an image than it does to apply a filter to it. ➔ You can also collect a certain number of items together and process them as a single batch.
- 66. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 67. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 68. Data Flow Visibility ➔ In a distributed system, getting an overall sense of where and how data flows can be a daunting task. ➔ Message queues, and routing rules, help identify and understand data flow paths. ➔ Through the rate with which they are processed, one can easily identify under- performing processes or areas where the data flow is not optimal.
- 70. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 71. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 72. Async Communication ➔ A lot of times, you don’t want to or need to process a message immediately. ➔ Message queues enable asynchronous processing, which allows you to put a message on the queue without processing it immediately. ➔ Queue up as many messages as you like, then process them at your leisure. ➔ Opens possibilities for retry-later setups.
- 73. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication
- 74. 10 benefits of using message queues Decoupling Redundancy Load Balancing, Scalability Elasticity & Spikability Resiliency Delivery Guarantees Ordering Guarantees Buffering Data Flow visibility Asynchronous communication http://www.iron.io/top-10-uses-for-message-queue/
- 75. RabbitMQ
- 76. RabbitMQ ➔ An AMQP messaging broker ➔ Written in Erlang ➔ Launched in 2007 ➔ Originally written by Rabbit Technologies Ltd. in London, UK ➔ Got acquired in 2010 by a division of VMWare ➔ Spin-off in 2013 to Pivotal Software, Inc ➔ Core participant of the AMQP working group
- 77. AMQP ➔ Advanced Message Queue Protocol ➔ An open standard, independent from RabbitMQ ➔ Provider agnostic, in theory. A client should be able to swap RabbitMQ with different AMQP server ➔ Client libraries available for all major programming languages ➔ Most recent specification version is 1.0. * * RabbitMQ mostly AMQP version 0.9.1 as of Feb 2016.
- 80. RabbitMQ server Basic concept Publisher Exchange
- 87. Publishing ➔ Your apps publish messages. Each message has ◆ routing key (e.g. subject) ◆ delivery mode (persistent or not persistent) ◆ headers (e.g. { "content-type": "application/json" }) ◆ payload (e.g. { "some": "json" }) ◆ properties (e.g. correlation-id)
- 91. Exchanges ➔ Exchanges are AMQP entities where messages are sent to. ➔ Exchanges on their own are useless, unless bound to something. https://www.rabbitmq.com/tutorials/amqp-concepts.html
- 92. Four types of exchanges ➔ Direct, Fanout, Topic, Headers ➔ Type defines how you can bind from it ➔ Most useful is Topic (allows regexp-like bindings) ➔ Read more from RabbitMQ docs
- 95. Bindings ➔ Rules with which messages are routed from exchanges to queues ➔ Each queue and exchange can have multiple bindings ➔ Examples: [characteristic].[color].[kind] ◆ *.orange.* ◆ *.*.rabbit ◆ lazy.#
- 96. Bindings Q1 is interested in all the orange animals. Q2 wants to hear everything about rabbits, and everything about lazy animals.
- 99. Queues ➔ Ordered lists of messages to be consumed ➔ FIFO ➔ Can be either durable or transient ➔ Can have arguments (behavioral properties) TTL of messages in it auto-expiry maximum length dead letter handling
- 102. Consuming ➔ Act of receiving and acting upon messages from a queue
- 103. Consuming ➔ Act of receiving and acting upon messages from a queue ➔ One at a time, or multiple-in-flight with acking
- 104. Consuming ➔ Act of receiving and acting upon messages from a queue ➔ One at a time, or multiple-in-flight with acking ➔ Delivery and execution guarantees
- 105. Consuming ➔ Act of receiving and acting upon messages from a queue ➔ One at a time, or multiple-in-flight with acking ➔ Delivery and execution guarantees ➔ One queue can have multiple consumers (enables load balancing — but then exact order is not guaranteed)
- 106. Consuming ➔ Act of receiving and acting upon messages from a queue ➔ One at a time, or multiple-in-flight with acking ➔ Delivery and execution guarantees ➔ One queue can have multiple consumers (enables load balancing — but then exact order is not guaranteed) ➔ A consumer may ask for exclusivity
- 107. Consumption sequence diagram RabbitMQ Consumer RabbitMQ Consumer [message]
- 108. Consumption sequence diagram RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’
- 109. Consumption sequence diagram RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the message
- 110. Consumption sequence diagram RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the messageack
- 111. Consumption sequence diagram RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the messageack mark message delivered
- 112. Consumption sequence diagram with a failure RabbitMQ Consumer RabbitMQ Consumer [message]
- 113. Consumption sequence diagram with a failure RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’
- 114. Consumption sequence diagram with a failure RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the message
- 115. Consumption sequence diagram with a failure RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the message dies!
- 116. Consumption sequence diagram with a failure RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the message (connection reset) dies!
- 117. Consumption sequence diagram with a failure RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the message (connection reset) dies! put message back to ‘ready’
- 118. Consumption sequence diagram with a failure RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the message (connection reset) dies! put message back to ‘ready’ Consumer Consumer
- 119. Consumption sequence diagram with a failure RabbitMQ Consumer RabbitMQ Consumer [message] mark message ‘unacked’ attempt to process the message (connection reset) dies! put message back to ‘ready’ Consumer Consumer [message] mark message ‘unacked’
- 120. Consumer characteristics (Node.js) ➔ Persistent TCP connection to AMQP server ➔ RabbitMQ pushes messages to consumer (no polling) ➔ Connection is multiplexed (using channels) ➔ Event listener/callback function is executed per each message, with ack() callback provided to be called when a message can be considered consumed.
- 125. RabbitMQ server Basic concept Deals service statistics queue webhook queue statistics service webhook service deals exchange webhook service Photos service photos exchange analyzer queue analyzer service resizing queue photo resizer service
- 126. So it’s like… a series of tubes.
- 128. Some like to call it “a post box, a post office and a postman, all in one”...
- 129. “When you send mail to the post box you're pretty sure that Mr. Postman will eventually deliver the mail to your recipient. Using this metaphor RabbitMQ is a post box, a post office and a postman.” From the RabbitMQ official tutorial.
- 130. However, in a microservices environment, this post metaphor can be dangerous.
- 131. The post metaphor can lure you into publishing messages with a specific consumer in mind, thus increasing tight coupling which makes it harder to add other kinds of consumers in the future without having to change multiple services.
- 133. User signs up Example use case shown by Sam Newman in “Principles of microservices” presentation
- 134. User signs up Create user account Example use case shown by Sam Newman in “Principles of microservices” presentation
- 135. User signs up Create user account Sign up to newsletter at MailChimp Example use case shown by Sam Newman in “Principles of microservices” presentation
- 136. User signs up Create user account Sign up to newsletter at MailChimp Send welcome email Example use case shown by Sam Newman in “Principles of microservices” presentation
- 137. User signs up Create user account Sign up to newsletter at MailChimp Send welcome email Send welcome gift Example use case shown by Sam Newman in “Principles of microservices” presentation
- 138. User signs up Create user account Sign up to newsletter at MailChimp Send welcome email Send welcome gift Example use case shown by Sam Newman in “Principles of microservices” presentation Done!
- 139. Orchestration
- 140. Example use case shown by Sam Newman in “Principles of microservices” presentation Sign up service Orchestration
- 141. Example use case shown by Sam Newman in “Principles of microservices” presentation Orchestration Sign up service Newsletter service subscribe
- 142. Example use case shown by Sam Newman in “Principles of microservices” presentation Orchestration Sign up service Newsletter service Email sendersubscribe send email
- 143. Example use case shown by Sam Newman in “Principles of microservices” presentation Orchestration Sign up service Newsletter service Email sender Delivery service subscribe send email send package
- 144. ➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind. Problem with orchestration — tight coupling
- 145. ➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind. ➔ Makes it hard to plug in new services along the way, and thus shape the system organically. Problem with orchestration — tight coupling
- 146. ➔ With orchestration, messages (commands, really) sent from service A to others end up being explicit and with a single recipient in mind. ➔ Makes it hard to plug in new services along the way, and thus shape the system organically. ➔ Expectations of replies to completed commands — seeing a “send this newsletter out” command in service A may seem like it is a synchronous action, whereas with a message queue it is not. Problem with orchestration — tight coupling
- 148. Choreography Sign up service Example use case shown by Sam Newman in “Principles of microservices” presentation
- 149. Example use case shown by Sam Newman in “Principles of microservices” presentation Choreography Sign up service user signed up
- 150. Example use case shown by Sam Newman in “Principles of microservices” presentation Choreography Sign up service Newsletter service user signed up
- 151. Example use case shown by Sam Newman in “Principles of microservices” presentation Choreography Sign up service Email sender Newsletter service user signed up
- 152. Example use case shown by Sam Newman in “Principles of microservices” presentation Choreography Sign up service Email sender Delivery service Newsletter service user signed up
- 153. Choreography Sign up service Newsletter service Email sender Delivery service RabbitMQ Queue Queue signups Queue Example use case shown by Sam Newman in “Principles of microservices” presentation
- 154. ➔ Responsive User does not have to wait for all services to complete their work, thus gets the response faster. Choreography
- 155. ➔ Responsive User does not have to wait for all services to complete their work, thus gets the response faster. ➔ Resilient Failure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed. Choreography
- 156. ➔ Responsive User does not have to wait for all services to complete their work, thus gets the response faster. ➔ Resilient Failure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed. ➔ Elastic Queue lengths can be used to scale consuming services up or down. Choreography
- 157. ➔ Responsive User does not have to wait for all services to complete their work, thus gets the response faster. ➔ Resilient Failure is tolerated in each service separately — if one fails then the work is queued and processing will eventually still happen, after error is removed. ➔ Elastic Queue lengths can be used to scale consuming services up or down. ➔ Works well with message queues. Allows reactive architectures. Choreography
- 160. Dumb service A Dumb service B Dumb service C Dumb service D Dumb service A Dumb service B Dumb service C Dumb service D Enterprise Service Bus
- 161. Dumb service A Dumb service B Dumb service C Dumb service D Dumb service A Dumb service B Dumb service C Dumb service D Magic Mystery Bus Example credit to Sam Newman in “Principles of microservices” presentation.
- 162. Dumb service A Dumb service B Dumb service C Dumb service D Dumb service A Dumb service B Dumb service C Dumb service D Magic Mystery Bus
- 163. Dumb pipes, smart endpoints ➔ Initially, you deploy an empty RabbitMQ.
- 164. Dumb pipes, smart endpoints ➔ Initially, you deploy an empty RabbitMQ. ➔ Do not make RabbitMQ deployment be aware of the desired data flows.
- 165. Dumb pipes, smart endpoints ➔ Initially, you deploy an empty RabbitMQ. ➔ Do not make RabbitMQ deployment be aware of the desired data flows. ➔ Exchanges, bindings and queues are created by services themselves as they need.
- 167. RabbitMQ server Basic concept political map Deals service statistics queue webhook queue # #.company_15 statistics service webhook service deals exchange webhook service
- 168. RabbitMQ server Basic concept political map Deals service statistics queue webhook queue # #.company_15 statistics service webhook service deals exchange webhook serviceKingdom of Deals Service Commonwealth of Statistics Service Republic of Webhook Service
- 169. So, in essence, RabbitMQ is left with nothing but the dumb pipes.
- 170. Smart service A Smart service B Smart service C Smart service D Smart service A Smart service B Smart service C Smart service D Dumb pipes
- 171. RabbitMQ in the wild
- 172. RabbitMQ at Pipedrive ➔ Started using in 2012
- 173. RabbitMQ at Pipedrive ➔ Started using in 2012 ➔ Main backbone of async service-to-service communication
- 174. RabbitMQ at Pipedrive ➔ Started using in 2012 ➔ Main backbone of async service-to-service communication ➔ All data change events are published to RabbitMQ.
- 175. RabbitMQ at Pipedrive ➔ Started using in 2012 ➔ Main backbone of async service-to-service communication ➔ All data change events are published to RabbitMQ. ➔ Averaging around >1,700 msg/sec. Peaking sometimes at 3,000+ msg/sec. >140M msg/day.
- 176. RabbitMQ at Pipedrive ➔ Started using in 2012 ➔ Main backbone of async service-to-service communication ➔ All data change events are published to RabbitMQ. ➔ Averaging around >1,700 msg/sec. Peaking sometimes at 3,000+ msg/sec. >140M msg/day.
- 178. RabbitMQ at Pipedrive ➔ Initially deployed a 3-node cluster MQ MQ MQ
- 179. RabbitMQ at Pipedrive ➔ Initially deployed a 3-node cluster ➔ Then scaled it up to a 5-node cluster MQ MQ MQ MQ MQ MQ MQ MQ
- 180. RabbitMQ at Pipedrive ➔ Initially deployed a 3-node cluster ➔ Then scaled it up to a 5-node cluster ➔ Then decided to move to multiple 3-node clusters instead MQ MQ MQ MQ MQ MQ MQ MQ MQ MQ MQ MQ MQ MQ
- 182. RabbitMQ at Pipedrive ➔ Started seeing networking overhead in a 5-node cluster as the load grew.
- 183. RabbitMQ at Pipedrive ➔ Started seeing networking overhead in a 5-node cluster as the load grew. ➔ Multiple clusters with semi-isolated traffic (bound by either SLA or logical service groups) was easier to handle for us.
- 184. RabbitMQ at Pipedrive ➔ Started seeing networking overhead in a 5-node cluster as the load grew. ➔ Multiple clusters with semi-isolated traffic (bound by either SLA or logical service groups) was easier to handle for us. ➔ It was a logical architecture decision, not a technical constraint — RabbitMQ does scale well beyond our load.
- 185. 3 caveats
- 188. Exchange ownership question ➔ Ultimately, every service will depend on existence of certain exchanges.
- 189. Exchange ownership question ➔ Ultimately, every service will depend on existence of certain exchanges. ➔ If these are missing, binding will throw an error.
- 190. Exchange ownership question ➔ Ultimately, every service will depend on existence of certain exchanges. ➔ If these are missing, binding will throw an error. ➔ But which service should create the exchange? There should ideally be only one owner per each exchange.
- 191. Exchange ownership question ➔ Ultimately, every service will depend on existence of certain exchanges. ➔ If these are missing, binding will throw an error. ➔ But which service should create the exchange? There should ideally be only one owner per each exchange. ➔ At Pipedrive, we have so far had shared exchange ownerships. Eventually it could create problems down the line as exchange properties are defined in multiple services.
- 192. Exchange ownership question ➔ Ultimately, every service will depend on existence of certain exchanges. ➔ If these are missing, binding will throw an error. ➔ But which service should create the exchange? There should ideally be only one owner per each exchange. ➔ At Pipedrive, we have so far had shared exchange ownerships. Eventually it could create problems down the line as exchange properties are defined in multiple services. ➔ Possible solution: die consumers when an exchange does not exist?
- 193. 2. Adding/removing bindings is like changing a database schema
- 194. Adding/removing bindings is like changing schema
- 195. Adding/removing bindings is like changing schema ➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema.
- 196. Adding/removing bindings is like changing schema ➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema. ➔ It takes time.
- 197. Adding/removing bindings is like changing schema ➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema. ➔ It takes time. ➔ At Pipedrive, we used to create a queue+binding for each logged in user to facilitate websocket connection back to end user’s browser.
- 198. Adding/removing bindings is like changing schema ➔ With high throughput, adding and removing bindings from a busy exchange can be like changing your DB schema. ➔ It takes time. ➔ At Pipedrive, we used to create a queue+binding for each logged in user to facilitate websocket connection back to end user’s browser. ➔ Don’t do it.
- 199. tl;dr: RabbitMQ is optimized for throughput of messages, not for throughput of schema changes on the fly.
- 200. At Pipedrive, we have a microservice we call socketqueue.
- 201. It’s a piping service between user’s browser and RabbitMQ websocket User’s browser socketqueue (consumer) RabbitMQ REST APIs (publishers) AMQPAMQP
- 202. Socketqueue 1.0 architecture ➔ One queue + binding per each connected user ➔ One Websocket connection per each connected user ➔ One-to-one relation between queue and websocket inside the socketqueue service. ➔ Horizontally scalable
- 203. RabbitMQ cluster Socketqueue 1.0 architecture REST APIs q3 api.events exchange q2 q1 socketq1 u1 u2 u3 q6 q5 q4 socketq2 u4 u5 u6 q8 socketq3 u7 u8 u9 q7 + 10K more q9 300+ msg/sec
- 204. RabbitMQ cluster Socketqueue 1.0 architecture REST APIs q3 api.events exchange q2 q1 socketq1 u1 u2 u3 q6 q5 q4 socketq2 u4 u5 u6 q8 socketq3 u7 u8 u9 q7 + 10K more q9 Did not scale beyond 10K queues, bindings 300+ msg/sec
- 205. RabbitMQ cluster Socketqueue 1.0 architecture REST APIs q3 api.events exchange q2 q1 socketq1 u1 u2 u3 q6 q5 q4 socketq2 u4 u5 u6 q8 socketq3 u7 u8 u9 q7 + 10K more q9 300+ msg/sec
- 206. Socketqueue 2.0 architecture REST APIs api.events exchange q-socket1 socketq1 u1 u2 u3 q-socket2 socketq2 u4 u5 u6 socketq3 u7 u8 u9 q-socket3 RabbitMQ cluster 300+ msg/sec
- 207. Socketqueue 1.0 architecture ➔ One queue + binding per each connected user ➔ One Websocket connection per each connected user ➔ One-to-one relation between queue and websocket inside the socketqueue service. ➔ Horizontally scalable
- 208. Socketqueue 1.02.0 architecture ➔ One queue + binding per each connected user socketqueue service instance. ➔ One Websocket connection per each connected user ➔ One-to-onemany relation between queue and websocket inside the socketqueue service. ➔ Horizontally scalable well beyond 10K simultaneous users
- 209. Most problematic? Initial setup of 10K+ queues/bindings during version upgrades, service restarts.
- 211. 100% delivery guarantee = Publisher Acknowledgements
- 212. Publisher Acknowledgements ➔ Your service sends a persistent message to RabbitMQ
- 213. Publisher Acknowledgements ➔ Your service sends a persistent message to RabbitMQ ➔ RabbitMQ receives your message but does not yet write it to disk
- 214. Publisher Acknowledgements ➔ Your service sends a persistent message to RabbitMQ ➔ RabbitMQ receives your message but does not yet write it to disk ➔ RabbitMQ tells you it has received your message
- 215. Publisher Acknowledgements ➔ Your service sends a persistent message to RabbitMQ ➔ RabbitMQ receives your message but does not yet write it to disk ➔ RabbitMQ tells you it has received your message ➔ The message gets routed to a durable queue
- 216. Publisher Acknowledgements ➔ Your service sends a persistent message to RabbitMQ ➔ RabbitMQ receives your message but does not yet write it to disk ➔ RabbitMQ tells you it has received your message ➔ The message gets routed to a durable queue ➔ A consumer picks up the message and starts processing it
- 217. Publisher Acknowledgements ➔ Your service sends a persistent message to RabbitMQ ➔ RabbitMQ receives your message but does not yet write it to disk ➔ RabbitMQ tells you it has received your message ➔ The message gets routed to a durable queue ➔ A consumer picks up the message and starts processing it ➔ RabbitMQ server dies and comes back up
- 218. Publisher Acknowledgements ➔ Your service sends a persistent message to RabbitMQ ➔ RabbitMQ receives your message but does not yet write it to disk ➔ RabbitMQ tells you it has received your message ➔ The message gets routed to a durable queue ➔ A consumer picks up the message and starts processing it ➔ RabbitMQ server dies and comes back up ➔ Except, that message that was being processed? It’s not there any more.
- 219. 100% delivery guarantee = Publisher Acknowledgements
- 220. Read more from https://www.rabbitmq.com/confirms.html
- 221. RabbitMQ elsewhere
- 223. 1 million messages per second
- 227. To put this into context...
- 228. 11B 2014 all of SMS traffic per day globally http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/ http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statistics http://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2
- 229. 17.2B 11B 2014 all of SMS traffic per day globally 2016 Apple iMessages per day globally http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/ http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statistics http://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2
- 230. 17.2B 11B 2014 all of SMS traffic per day globally 2016 Apple iMessages per day globally Facebook, Messenger, and WhatsApp per day combined 45B http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/ http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statistics http://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2
- 231. 86.4B 17.2B 11B 2014 all of SMS traffic per day globally 2016 Apple iMessages per day globally That single RabbitMQ experiment Facebook, Messenger, and WhatsApp per day combined 45B http://techcrunch.com/2015/04/22/facebook-voip-not-facebook-phone/ http://www.openuniversity.edu/news/news/2014-text-messaging-usage-statistics http://www.businessinsider.com/eddy-cue-200k-imessages-per-second-2016-2
- 232. I think that’s pretty remarkable.
- 234. Thank you. Martin Tajur @tajur on Twitter With this promo code DEVCLUBEE 2 months free www.pipedrive.com
- 235. Thank you. Martin Tajur @tajur on Twitter With this promo code DEVCLUBEE 2 months free www.pipedrive.com We’re hiring.
- 236. Questions/Answers