Do you need to process thousands of images in the background for your web app? Do you need to share data across multiple applications, probably written in different languages and sitting at different servers? Your real time data feed is becoming slow because you are polling the database constantly for new data updates? Do you need to scale information processing during peek times? What about deploying new features with zero downtime? If any of these problems sound familiar then you probably need to use messaging in your application. In this talk I will introduce RabbitMQ, a messaging and queue server that can help us tackle those problems. We will learn the benefits of a Queue Server and see how to integrate messaging into our applications. With this talk we hope that the term 'decoupling' gets a new, broader, meaning.

1. Scaling Applications
with RabbitMQ
Alvaro Videla - RabbitMQ

2. Alvaro Videla
•
•
•
•
•
Developer Advocate at Pivotal / RabbitMQ!
Co-Author of RabbitMQ in Action!
Creator of the RabbitMQ Simulator!
Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!
@old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

3. About Me
Co-authored!
!
RabbitMQ in Action!
http://bit.ly/rabbitmq

4. About this Talk
•
Exploratory Talk
•
A ‘what could be done’ talk instead of ‘this is how you do it’

5. What is RabbitMQ

6. RabbitMQ

7. RabbitMQ

8. RabbitMQ
• Multi Protocol Messaging Server

9. RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)

10. RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot

11. RabbitMQ
• Multi Protocol Messaging Server!
• Open Source (MPL)!
• Polyglot!
• Written in Erlang/OTP

12. Multi Protocol
http://bit.ly/rmq-protocols

13. Polyglot

14. Polyglot

15. Polyglot
• Java

16. Polyglot
• Java!
• node.js

17. Polyglot
• Java!
• node.js!
• Erlang

18. Polyglot
• Java!
• node.js!
• Erlang!
• PHP

19. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby

20. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net

21. Polyglot
• Java!
• node.js!
• Erlang!
• PHP!
• Ruby!
• .Net!
• Haskell

22. Polyglot
Even COBOL!!!11

23. Some users of RabbitMQ

24. Some users of RabbitMQ
•
Instagram

25. Some users of RabbitMQ
•
•
Instagram!
Indeed.com

26. Some users of RabbitMQ
•
•
•
Instagram!
Indeed.com!
Telefonica

27. Some users of RabbitMQ
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre

28. Some users of RabbitMQ
•
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre!
NHS

29. Some users of RabbitMQ
•
•
•
•
•
•
Instagram!
Indeed.com!
Telefonica!
Mercado Libre!
NHS!
Mozilla

30. http://www.rabbitmq.com/download.html
Unix - Mac - Windows

31. Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator

32. http://tryrabbitmq.com

33. RabbitMQ Simulator

34. What you get out of the box

35. Some RabbitMQ Features

36. Some RabbitMQ Features
•
No message loss

37. Some RabbitMQ Features
•
No message loss
•
Persistent Messages

38. Some RabbitMQ Features
•
No message loss
•
Persistent Messages
•
Publisher Confirms

39. Some RabbitMQ Features
•
No message loss
•
Persistent Messages
•
Publisher Confirms
•
Message Acknowledgment

40. Some RabbitMQ Features
•
No message loss
•
Persistent Messages
•
Publisher Confirms
•
Message Acknowledgment
•
Mirrored Queues

41. Some RabbitMQ Features
•
No message loss
•
Persistent Messages
•
Publisher Confirms
•
Message Acknowledgment
•
Mirrored Queues
•
Message Ordering

42. Some RabbitMQ Features

43. Some RabbitMQ Features
•
Dead Letter Exchanges

44. Some RabbitMQ Features
•
Dead Letter Exchanges
•
Alternate Exchanges

45. Some RabbitMQ Features
•
Dead Letter Exchanges
•
Alternate Exchanges
•
Message and Queues TTLs

46. Some RabbitMQ Features
•
Dead Letter Exchanges
•
Alternate Exchanges
•
Message and Queues TTLs
•
Consumer Priorities

47. Some RabbitMQ Features
•
Dead Letter Exchanges
•
Alternate Exchanges
•
Message and Queues TTLs
•
Consumer Priorities
•
Federation

48. Some RabbitMQ Features
•
Dead Letter Exchanges
•
Alternate Exchanges
•
Message and Queues TTLs
•
Consumer Priorities
•
Federation
•
Shovel

49. Preventing unbounded
buffers with RabbitMQ
http://bit.ly/1eCKYK8

50. RabbitMQ Management

51. Scenario
Batch Processing

52. Requirements

53. Requirements
•
Generate XML

54. Requirements
•
Generate XML
•
Distribution Over a Cluster

55. Requirements
•
Generate XML
•
Distribution Over a Cluster
•
Elasticity - Add/Remove new workers

56. Requirements
•
Generate XML
•
Distribution Over a Cluster
•
Elasticity - Add/Remove new workers
•
No Code Changes

57. Design

58. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
!
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'video-desc-ex');
!
$channel->close();
$conn->close();

59. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
!
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'video-desc-ex');
!
$channel->close();
$conn->close();

60. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
!
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'video-desc-ex');
!
$channel->close();
$conn->close();

61. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
!
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'video-desc-ex');
!
$channel->close();
$conn->close();

62. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
!
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'video-desc-ex');
!
$channel->close();
$conn->close();

63. Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
!
$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'video-desc-ex');
!
$channel->close();
$conn->close();

64. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
!
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

65. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
!
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

66. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
!
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

67. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
!
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

68. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
!
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

69. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
!
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

70. Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);
$channel = $conn->channel();
!
$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);
$channel->queue_declare('video-desc-queue', false, true,
false, false);
$channel->queue_bind('video-desc-queue', 'video-desc-ex');
!
$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

71. Scenario
Upload Pictures

72. Requirements

73. Requirements
•
Upload Picture

74. Requirements
•
Upload Picture
•
Reward User

75. Requirements
•
Upload Picture
•
Reward User
•
Notify User Friends

76. Requirements
•
Upload Picture
•
Reward User
•
Notify User Friends
•
Resize Picture

77. Requirements
•
Upload Picture
•
Reward User
•
Notify User Friends
•
Resize Picture
•
No Code Changes

78. Design

79. Design

80. Design

81. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
!
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
!
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'upload-pictures');

82. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
!
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
!
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'upload-pictures');

83. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
!
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
!
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'upload-pictures');

84. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
!
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
!
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'upload-pictures');

85. Publisher Code
$channel->exchange_declare('upload-pictures', 'fanout', false,
true, false);
!
$metadata = json_encode(array(
'image_id' => $image_id,
'user_id' => $user_id,
‘image_path' => $image_path));
!
$msg = new AMQPMessage($metadata, array('content_type' =>
'application/json',
'delivery_mode' => 2));
!
$channel->basic_publish($msg, 'upload-pictures');

86. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
!
$channel->queue_declare('resize-picture', false, true,
false, false);
!
$channel->queue_bind('resize-picture', 'upload-pictures');
!
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

87. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
!
$channel->queue_declare('resize-picture', false, true,
false, false);
!
$channel->queue_bind('resize-picture', 'upload-pictures');
!
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

88. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
!
$channel->queue_declare('resize-picture', false, true,
false, false);
!
$channel->queue_bind('resize-picture', 'upload-pictures');
!
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

89. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
!
$channel->queue_declare('resize-picture', false, true,
false, false);
!
$channel->queue_bind('resize-picture', 'upload-pictures');
!
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

90. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
!
$channel->queue_declare('resize-picture', false, true,
false, false);
!
$channel->queue_bind('resize-picture', 'upload-pictures');
!
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

91. Consumer Code
$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);
!
$channel->queue_declare('resize-picture', false, true,
false, false);
!
$channel->queue_bind('resize-picture', 'upload-pictures');
!
$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);
!
while(count($channel->callbacks)) {
$channel->wait();
}

92. Consumer Code
$consumer = function($msg){
!
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};

93. Consumer Code
$consumer = function($msg){
!
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};

94. Consumer Code
$consumer = function($msg){
!
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};

95. Consumer Code
$consumer = function($msg){
!
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};

96. Consumer Code
$consumer = function($msg){
!
$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->
basic_ack($msg->delivery_info['delivery_tag']);
};

97. Scenario
Distributed Logging

98. Requirements

99. Requirements
•
Several Web Servers

100. Requirements
•
Several Web Servers
•
Logic Separated by Module/Action

101. Requirements
•
Several Web Servers
•
Logic Separated by Module/Action
•
Several Log Levels: info, warning and error

102. Requirements
•
Several Web Servers
•
Logic Separated by Module/Action
•
Several Log Levels: info, warning and error
•
Add Remove log listeners at will

103. Design

104. Design

105. Design

106. Design

107. Design

108. Publisher Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));
!
$channel->basic_publish($msg, 'logs',
'server1.user.profile.info');

109. Publisher Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));
!
$channel->basic_publish($msg, 'logs',
'server1.user.profile.info');

110. Publisher Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));
!
$channel->basic_publish($msg, 'logs',
'server1.user.profile.info');

111. Publisher Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));
!
$channel->basic_publish($msg, 'logs',
'server1.user.profile.info');

112. Consumer Code
Get messages sent by host:
!
server1

113. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$channel->queue_declare('server1-logs', false, true,
false, false);
!
$channel->queue_bind('server1-logs', 'logs', 'server1.#');

114. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$channel->queue_declare('server1-logs', false, true,
false, false);
!
$channel->queue_bind('server1-logs', 'logs', 'server1.#');

115. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$channel->queue_declare('server1-logs', false, true,
false, false);
!
$channel->queue_bind('server1-logs', 'logs', 'server1.#');

116. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$channel->queue_declare('server1-logs', false, true,
false, false);
!
$channel->queue_bind('server1-logs', 'logs', 'server1.#');

117. Consumer Code
Get all error messages

118. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$channel->queue_declare('error-logs', false, true,
false, false);
!
$channel->queue_bind('error-logs', 'logs', '#.error');

119. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$channel->queue_declare('error-logs', false, true,
false, false);
!
$channel->queue_bind('error-logs', 'logs', '#.error');

120. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$channel->queue_declare('error-logs', false, true,
false, false);
!
$channel->queue_bind('error-logs', 'logs', '#.error');

121. Consumer Code
$channel->exchange_declare('logs', 'topic', false,
true, false);
!
$channel->queue_declare('error-logs', false, true,
false, false);
!
$channel->queue_bind('error-logs', 'logs', '#.error');

122. Distributed Apps

123. Distributed Application
App
App
App
App

124. Distributed Application
App
App
App
App

125. Ad-hoc solution

126. A process that replicates data
to the remote server

127. Possible issues
•
Remote server is offline
•
•
•
Prevent unbounded local buffers
Prevent message loss
Prevent unnecessary message replication
•
No need for those messages on remote server
•
Messages that became stale

128. Can we do better?

129. RabbitMQ Federation

130. RabbitMQ Federation
•
Supports replication across different administrative domains
•
Supports mix of Erlang and RabbitMQ versions
•
Supports Network Partitions
•
Specificity - not everything has to be federated

131. RabbitMQ Federation

132. RabbitMQ Federation

133. RabbitMQ Federation

134. RabbitMQ Federation
•
It’s a RabbitMQ Plugin

135. RabbitMQ Federation
•
It’s a RabbitMQ Plugin
•
Internally uses Queues and Exchanges Decorators

136. RabbitMQ Federation
•
It’s a RabbitMQ Plugin
•
Internally uses Queues and Exchanges Decorators
•
Managed using Parameters and Policies

137. Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation

138. Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management

139. Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'

140. Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream
‘{“uri":"amqp://server-name","expires":3600000}'
!
rabbitmqctl set_policy --apply-to exchanges federate-me "^amq."
'{"federation-upstream-set":"all"}'

141. Federating an Exchange

142. With RabbitMQ we can

143. With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP

144. With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation

145. With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation
•
Scale up using with different consumer strategies

146. With RabbitMQ we can
•
Ingest data using various protocols: AMQP, MQTT and STOMP
•
Distribute that data globally using Federation
•
Scale up using with different consumer strategies
•
Integrate across many platforms and programming languages

147. Questions?

148. Talk and Slides
https://joind.in/talk/view/10525

149. Thanks
Alvaro Videla - @old_sound
Credits
world map: wikipedia.org
federation diagrams: rabbitmq.com