Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Message queueing

I

  • Login to see the comments

Message queueing

  1. 1. Message Queueing from an MQ noobs perspective
  2. 2. Preamble• I work for a global telco• We push telco events and messages around• All should be processed quickly and reliably• Some generate a response
  3. 3. Scenario• We send messages around in small text files, sometimes with RFC 2822 headers• Use the UNIX sendf/recfiled mechanism
  4. 4. Problem• Its entirely intolerant to faults• ... and network latency
  5. 5. Our Solution• Add in our own "message queueing"• ... and some facilities to decouple delays
  6. 6. Shortcomings• Messages cannot be sent to multiple hosts• Target hosts are not easily reconfigurable• Its developed in-house and has issues
  7. 7. Scenario• multiple producers• a consumer and its backup• Systems separated by 100-400ms WAN • with occasional 100% packet loss
  8. 8. The Contenders
  9. 9. pymq• very hazy on the details• depends on MySQL? and Django?!?• ignored while looking at kombu / rabbitmq
  10. 10. ZeroMQ aka 0MQ"ZeroMQ is a message orientated IPC Library." - commenter on stackoverflow
  11. 11. AMQP• Is a standard for message queueing• Producers submit messages to brokers• Brokers consist of exchanges and queues• Exchanges route the messages• Queues store messages• Consumers pull messages out
  12. 12. RabbitMQ
  13. 13. RabbitMQProducer Broker Consumer
  14. 14. RabbitMQProducer Broker Exchange Queue Consumer
  15. 15. RabbitMQQueues
  16. 16. RabbitMQExchanges
  17. 17. RabbitMQ
  18. 18. RabbitMQExchanges
  19. 19. RabbitMQ Routing
  20. 20. RabbitMQExchange Types
  21. 21. RabbitMQDirect Exchanges
  22. 22. RabbitMQTopic Exchanges
  23. 23. RabbitMQFanout Exchanges
  24. 24. RabbitMQPersistence
  25. 25. RabbitMQBroker Persistence
  26. 26. RabbitMQMessage Persistence
  27. 27. RabbitMQ1. Mark the exchange “durable”.2. Mark the queue “durable”.3. Set the message’s “delivery mode” to a value of 2
  28. 28. RabbitMQPersistence Warnings
  29. 29. My Desired Setup Remote Host Processing Server Publisher Queue ebs Interw Queue Consumer
  30. 30. My Desired Setup Remote Host Processing Server Publisher Queue ebs Interw Queue Consumer Consumer
  31. 31. My Desired Setup Remote Host Processing Server Publisher Consumer ebs Interw Queue
  32. 32. ShovelRemote Host Processing Server Publisher Queue bs Queue rwe Consumer Inte Shovel
  33. 33. =ERROR REPORT==== 7-Aug-2012::16:50:32 ===** Generic server <0.366.0> terminating** Last message in was {$gen_cast,init}** When Server state == {state,undefined,undefined,undefined,undefined, hello_shovel, {shovel, {endpoint, [{amqp_params_direct,<<"guest">>,<<"/">>, rabbit@localhost,none,[]}], []}, {endpoint, [{amqp_params_network,<<"guest">>,<<"guest">>, <<>>,"hardrock",undefined,0,0,0,infinity,none, [#Fun<amqp_auth_mechanisms.plain.3>, #Fun<amqp_auth_mechanisms.amqplain.3>], [],[]}], []}, 0,on_confirm,#Fun<rabbit_shovel_sup.16.91344538>, #Fun<rabbit_shovel_sup.16.91344538>,<<"hello">>,5}, undefined,undefined,undefined,undefined,undefined}** Reason for termination ==** {{badmatch,{error,access_refused}}, [{rabbit_shovel_worker,make_conn_and_chan,1,[]}, {rabbit_shovel_worker,handle_cast,2,[]}, {gen_server2,handle_msg,2,[]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
  34. 34. error in config file "/usr/local/etc/rabbitmq/rabbitmq.config" (37): syntax errorbefore: .
  35. 35. Shovel[ {rabbitmq_shovel, [ {shovels, [ {hello_shovel, [ {sources, [ {brokers, ["amqp://"]} ]} , {destinations, [ {brokers, ["amqp://server"]} ]} , {queue, <<"hello">>} ]} ]} ]}].
  36. 36. [ Shovel {rabbitmq_shovel, [ {shovels, [ {hello_shovel, [ {sources, [ {brokers, ["amqp://"]} , {declarations, [ {exchange.declare, [ {exchange, <<"foo">>} , {type, <<"direct">>} , durable ]} , {queue.declare, [ {queue, <<"hello">>} ]} , {queue.bind, [ {exchange, <<"foo">>} , {queue, <<"hello">>} ]} ]} ]} , {destinations, [ {brokers, ["amqp://hardrock"]} ]} , {queue, <<"hello">>} ]} ]} ]}].
  37. 37. Installation• 1. install RabbitMQ • (install homebrew; brew install rabbitmq; rabbitmq-server [-detached]) • (yum install rabbitmq-server.noarch) • rabbitmqctl or http://server:55672/• 2. pip install kombu, or pika, or puka, or ...
  38. 38. pika"a pure-Python implementation of the AMQP 0-9-1 protocol"
  39. 39. Publisherimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)channel.basic_publish(exchange=, routing_key=hello, body=Hello World!, properties=pika.BasicProperties(delivery_mode=2))print " [x] Sent Hello World!"connection.close() Publisher Queue
  40. 40. Consumerimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(localhost))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,)channel.basic_consume(callback, queue=hello)channel.start_consuming() Queue Consumer
  41. 41. Remote Consumer Local Publisher
  42. 42. Remote queuesimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,)channel.basic_consume(callback, queue=hello)channel.start_consuming() Host 1 Host 2 Publisher Queue Consumer Consumer
  43. 43. Remote Consumer Local Consumer Local Publisher
  44. 44. Issues• pikas BlockingConnection (my use case is simple enough) hard-codes the socket timeout and fails to cope with latency >1s• Fails to cope at all with packet loss
  45. 45. Deleting a Queueimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_delete(queue=hello)channel.close()connection.close()
  46. 46. Kombu"a messaging framework for Python"
  47. 47. Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn: with conn.SimpleQueue(hello) as queue: queue.put(Hello World!) print " [x] Sent Hello World!"
  48. 48. Bizarro errorsamqplib.client_0_8.exceptions.AMQPChannelException: (406,u"PRECONDITION_FAILED - parameters for queue hello in vhost /not equivalent", (50, 10), Channel.queue_declare)
  49. 49. Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost//) as conn: with conn.SimpleQueue(hello, queue_opts=dict(durable=False)) as queue: queue.put(Hello World!) print " [x] Sent Hello World!"
  50. 50. Puka"An opinionated RabbitMQ client"
  51. 51. Publisherimport pukaclient = puka.Client(amqp://localhost//)client.wait(client.connect())client.wait(client.queue_declare(queue=hello, durable=True))promise = client.basic_publish(exchange=, routing_key=hello, body=Hello World!)client.wait(promise)print " [x] Sent Hello World!"client.wait(client.close())
  52. 52. Other Plugins• Management (web interface, quite nice)• Federation (Shovel, but with MOAR)• STOMP protocol support
  53. 53. STOMPSimple (or Streaming) Text Orientated Messaging Protocol
  54. 54. STOMP "simple"
  55. 55. STOMP{rabbitmq_stomp, [ {tcp_listeners, [ {"127.0.0.1", 61613} , {"::1", 61613} ]} , {default_user, [ {login, "guest"} , {passcode, "guest"} ]}]}
  56. 56. stompclientfrom stompclient import PublishClientclient = PublishClient(localhost, 61613)client.connect()client.send(/queue/hello, Hello, world!)client.disconnect()
  57. 57. stompclientfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn: with conn.SimpleQueue(hello) as queue: queue.put(Hello World!) print " [x] Sent Hello World!" from stompclient import PublishClient client = PublishClient(localhost, 61613) client.connect() client.send(/queue/hello, Hello, world!) client.disconnect()
  58. 58. stompclient from stompclient import PublishClient client = PublishClient(localhost, 61613) client.connect() client.send(/queue/hello, Hello, world!) client.disconnect()from stompclient import PublishClientwith PublishClient(localhost, 61613) as client: client.send(/queue/hello, Hello, world!)
  59. 59. Celery"a synchronous or asynchronous task queue/job queue based on distributed message passing"
  60. 60. TcpCatcher• TCP, SOCKS, HTTP(S) proxy & monitor• Can introduce latency and transmission errors• Understands HTTP and images• Can debug/interfere/log SSL traffic• Free: www.tcpcatcher.fr

×