This presentation was given at PyCon AU 2012 but not recorded. It was written as I learned about modern message queueing methods (in particular RabbitMQ.)

1. Message Queueing
from an MQ noob's perspective

2. Preamble
• I work for a global telco
• We push telco events and messages around
• All should be processed quickly and reliably
• Some generate a response

3. Scenario
• We send messages around in small text
files, sometimes with RFC 2822 headers
• Use the UNIX sendf/recfiled mechanism

5. Problem
• It's entirely intolerant to faults
• ... and network latency

6. Our Solution
• Add in our own "message queueing"
• ... and some facilities to decouple delays

7. Shortcomings
• Messages cannot be sent to multiple hosts
• Target hosts are not easily reconfigurable
• It's developed in-house and has issues

8. Scenario
• multiple producers
• a consumer and its backup
• Systems separated by 100-400ms WAN
• with occasional 100% packet loss

9. The Contenders

10. pymq
• very hazy on the details
• depends on MySQL? and Django?!?
• ignored while looking at kombu / rabbitmq

11. ZeroMQ aka 0MQ
"ZeroMQ is a message orientated IPC Library."
- commenter on stackoverflow

12. AMQP
• Is a standard for message queueing
• Producers submit messages to brokers
• Brokers consist of exchanges and queues
• Exchanges route the messages
• Queues store messages
• Consumers pull messages out

14. RabbitMQ

15. RabbitMQ
Producer
Broker
Consumer

16. RabbitMQ
Producer
Broker
Exchange
Queue
Consumer

17. RabbitMQ
Queues

18. RabbitMQ
Exchanges

19. RabbitMQ

20. RabbitMQ
Exchanges

21. RabbitMQ
Routing

22. RabbitMQ
Exchange Types

23. RabbitMQ
Direct Exchanges

24. RabbitMQ
Topic Exchanges

25. RabbitMQ
Fanout Exchanges

26. RabbitMQ
Persistence

27. RabbitMQ
Broker Persistence

28. RabbitMQ
Message Persistence

29. RabbitMQ
1. Mark the exchange “durable”.
2. Mark the queue “durable”.
3. Set the message’s “delivery mode” to a value of 2

30. RabbitMQ
Persistence Warnings

31. RabbitMQ
Acknowledging

32. My Desired Setup
Remote Host Processing Server
Publisher Queue
ebs
Interw
Queue Consumer

33. My Desired Setup
Remote Host Processing Server
Publisher Queue
ebs
Interw
Queue Consumer
Consumer

34. My Desired Setup
Remote Host Processing Server
Publisher Consumer
ebs
Interw
Queue

35. Shovel
Remote Host Processing Server
Publisher Queue
bs
Queue rwe Consumer
Inte
Shovel

36. =ERROR REPORT==== 7-Aug-2012::16:50:32 ===
** Generic server <0.366.0> terminating
** Last message in was {'$gen_cast',init}
** When Server state == {state,undefined,undefined,undefined,undefined,
hello_shovel,
{shovel,
{endpoint,
[{amqp_params_direct,<<"guest">>,<<"/">>,
rabbit@localhost,none,[]}],
[]},
{endpoint,
[{amqp_params_network,<<"guest">>,<<"guest">>,
<<>>,"hardrock",undefined,0,0,0,infinity,none,
[#Fun<amqp_auth_mechanisms.plain.3>,
#Fun<amqp_auth_mechanisms.amqplain.3>],
[],[]}],
[]},
0,on_confirm,#Fun<rabbit_shovel_sup.16.91344538>,
#Fun<rabbit_shovel_sup.16.91344538>,<<"hello">>,5},
undefined,undefined,undefined,undefined,undefined}
** Reason for termination ==
** {{badmatch,{error,access_refused}},
[{rabbit_shovel_worker,make_conn_and_chan,1,[]},
{rabbit_shovel_worker,handle_cast,2,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}

37. error in config file "/usr/local/etc/rabbitmq/rabbitmq.config" (37): syntax error
before: '.'

38. Shovel
[
{rabbitmq_shovel, [
{shovels, [
{hello_shovel, [
{sources, [
{brokers, ["amqp://"]}
]}
, {destinations, [
{brokers, ["amqp://server"]}
]}
, {queue, <<"hello">>}
]}
]}
]}
].

39. [
Shovel
{rabbitmq_shovel, [
{shovels, [
{hello_shovel, [
{sources, [
{brokers, ["amqp://"]}
, {declarations, [
{'exchange.declare', [
{exchange, <<"foo">>}
, {type, <<"direct">>}
, durable
]}
, {'queue.declare', [
{queue, <<"hello">>}
]}
, {'queue.bind', [
{exchange, <<"foo">>}
, {queue, <<"hello">>}
]}
]}
]}
, {destinations, [
{brokers, ["amqp://hardrock"]}
]}
, {queue, <<"hello">>}
]}
]}
]}
].

40. Installation
• 1. install RabbitMQ
• (install homebrew; brew install rabbitmq;
rabbitmq-server [-detached])
• (yum install rabbitmq-server.noarch)
• rabbitmqctl or http://server:55672/
• 2. pip install kombu, or pika, or puka, or ...

41. pika
"a pure-Python implementation of the
AMQP 0-9-1 protocol"

42. Publisher
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('server'))
channel = connection.channel()
channel.queue_declare(queue='hello', durable=True)
channel.basic_publish(exchange='', routing_key='hello', body='Hello World!',
properties=pika.BasicProperties(delivery_mode=2))
print " [x] Sent 'Hello World!'"
connection.close()
Publisher Queue

43. Consumer
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello', durable=True)
print ' [*] Waiting for messages. To exit press CTRL+C'
def callback(ch, method, properties, body):
print ' [x] Received %r' % (body,)
channel.basic_consume(callback, queue='hello', no_ack=True)
channel.start_consuming()
Queue Consumer

44. Remote Consumer Local Publisher

45. Remote queues
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('server'))
channel = connection.channel()
channel.queue_declare(queue='hello', durable=True)
print ' [*] Waiting for messages. To exit press CTRL+C'
def callback(ch, method, properties, body):
print ' [x] Received %r' % (body,)
channel.basic_consume(callback, queue='hello', no_ack=True)
channel.start_consuming()
Host 1 Host 2
Publisher Queue
Consumer Consumer

46. Remote Consumer Local Consumer Local Publisher

47. Acknowledgement
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('server'))
channel = connection.channel()
channel.queue_declare(queue='hello', durable=True)
print ' [*] Waiting for messages. To exit press CTRL+C'
def callback(ch, method, properties, body):
print ' [x] Received %r' % (body,)
raw_input('Press enter to ACK')
ch.basic_ack(delivery_tag=method.delivery_tag)
channel.basic_consume(callback, queue='hello', no_ack=False)
channel.start_consuming()

48. Consumer Publisher

49. Issues
• pika's BlockingConnection (my use case is
simple enough) hard-codes the socket
timeout and fails to cope with latency >1s
• Fails to cope at all with packet loss

50. Deleting a Queue
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('server'))
channel = connection.channel()
channel.queue_delete(queue='hello')
channel.close()
connection.close()

51. Kombu
"a messaging framework for Python"

52. Publisher
from kombu import BrokerConnection
with BrokerConnection('amqp://localhost') as conn:
with conn.SimpleQueue('hello') as queue:
queue.put('Hello World!')
print " [x] Sent 'Hello World!'"

53. Bizarro errors
amqplib.client_0_8.exceptions.AMQPChannelException: (406,
u"PRECONDITION_FAILED - parameters for queue 'hello' in vhost '/'
not equivalent", (50, 10), 'Channel.queue_declare')

54. Publisher
from kombu import BrokerConnection
with BrokerConnection('amqp://localhost//') as conn:
with conn.SimpleQueue('hello', queue_opts=dict(durable=False)) as queue:
queue.put('Hello World!')
print " [x] Sent 'Hello World!'"

55. Puka
"An opinionated RabbitMQ client"

56. Publisher
import puka
client = puka.Client('amqp://localhost//')
client.wait(client.connect())
client.wait(client.queue_declare(queue='hello', durable=True))
promise = client.basic_publish(exchange='', routing_key='hello',
body='Hello World!')
client.wait(promise)
print " [x] Sent 'Hello World!'"
client.wait(client.close())

57. Other Plugins
• Management (web interface, quite nice)
• Federation (Shovel, but with MOAR)
• STOMP protocol support

58. STOMP
Simple (or Streaming) Text Orientated
Messaging Protocol

59. STOMP
"simple"

60. STOMP
{rabbitmq_stomp, [
{tcp_listeners, [
{"127.0.0.1", 61613}
, {"::1", 61613}
]}
, {default_user, [
{login, "guest"}
, {passcode, "guest"}
]}
]}

61. stompclient
from stompclient import PublishClient
client = PublishClient('localhost', 61613)
client.connect()
client.send('/queue/hello', 'Hello, world!')
client.disconnect()

62. stompclient
from kombu import BrokerConnection
with BrokerConnection('amqp://localhost') as conn:
with conn.SimpleQueue('hello') as queue:
queue.put('Hello World!')
print " [x] Sent 'Hello World!'"
from stompclient import PublishClient
client = PublishClient('localhost', 61613)
client.connect()
client.send('/queue/hello', 'Hello, world!')
client.disconnect()

63. stompclient
from stompclient import PublishClient
client = PublishClient('localhost', 61613)
client.connect()
client.send('/queue/hello', 'Hello, world!')
client.disconnect()
from stompclient import PublishClient
with PublishClient('localhost', 61613) as client:
client.send('/queue/hello', 'Hello, world!')

64. Celery
"a synchronous or asynchronous task queue/job
queue based on distributed message passing"

65. TcpCatcher
• TCP, SOCKS, HTTP(S) proxy & monitor
• Can introduce latency and transmission
errors
• Understands HTTP and images
• Can debug/interfere/log SSL traffic
• Free: www.tcpcatcher.fr