The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP)

The Code of Forking Paths
Karel Minařík

Karel Minařík
→ Independent web designer and developer

→ Ruby, Rails, Git and CouchDB propagandista in .cz

→ Previously: Flash Developer; Art Director; Information Architect;… (see LinkedIn)

→ @karmiq at Twitter

→ karmi.cz


“He believed in an inﬁnite series of times, in a growing,
dizzying net of divergent, convergent and parallel
times. (...) We do not exist in the majority of these
times; in some you exist, and not I; in others I, and not
you; in others, both of us. In the present one, which
a favorable fate has granted me, you have arrived at my
house; in another, while crossing the garden, you found
me dead; in still another, I utter these same words, but
I am a mistake, a ghost.”

http://en.wikipedia.org/wiki/
The_Garden_of_Forking_Paths


THE CODE OF FORKING PATHS

Parallel presence

HTTP/1.1 503 Service Unavailable
HTTP/1.1 201 Created



Ataxo Social Insider


1 Asynchronous Task Processing



Asynchronous Task Processing

Long running requests

Canonical Example: video upload and conversion


Request WORKLOAD Response

Request NOTIFICATION Response

WORKLOAD



„Engineering“ solution

class UploadController < ApplicationController

def create
# ....
Thread.new do
# *** WORK REALLY HARD <HERE> *** What could possibly go wrong?
end
render :text => "KTHXBAI!"
end

end



The Task Queue

class UploadController < ApplicationController

def create
# Store uploaded file, store record
put_the_video_on_the_processing_queue(@video.id)
render :text => "Thanks! Your video is being processed."
end

end


REDIS

How Does It Work?

RPUSH

}
LPOP

O(1) Millions of items

https://github.com/defunkt/resque/blob/v1.13.0/lib/resque.rb#L133-138

http://redis.io/commands#list The Code of Forking Paths

TASK QUEUES

Poor-man’s Queues

1 „Publisher”
/usr/local/bin/redis-‐cli RPUSH "queue" "task-‐01"

2 „Worker”
while true; do /usr/local/bin/redis-‐cli BLPOP "queue" 0; done


TASK QUEUES

Poor-man’s Queues

1 „Publisher”
/usr/local/bin/redis-‐cli RPUSH "queue" "task-‐01"

2 „Worker”
while true; do
/usr/local/bin/redis-‐cli BLPOP "queue" 0
/usr/local/bin/redis-‐cli PUBLISH "queue:messages" "Processed task."
done

3 „Monitor”
/usr/local/bin/redis-‐cli SUBSCRIBE "queue:messages"

Demo

require "redis"
require "nest"

module Ost
VERSION = "0.0.1"
TIMEOUT = ENV["OST_TIMEOUT"] || 2

class Queue
attr :ns

def initialize(name)
@ns = Nest.new(:ost)[name]
end

def push(value)
redis.lpush(ns, value)
end

def each(&block)
loop do
_, item = redis.brpop(ns, TIMEOUT)
next if item.nil? or item.empty?

begin
block.call(item)
rescue Exception => e
error = "#{Time.now} #{ns[item]} => #{e.inspect}"

redis.rpush ns[:errors], error
redis.publish ns[:errors], error
end
end
end

def errors
redis.lrange ns[:errors], 0, -1
end

alias << push
alias pop each

private

def redis
Ost.redis
end
end

@queues = Hash.new do |hash, key|
hash[key] = Queue.new(key)
end

def self.[](queue)
@queues[queue]
end

def self.connect(options = {})
@redis = Redis.connect(options)
end

def self.redis
@redis ||= Redis.connect
end

def self.redis=(redis)
@redis = redis
end
end

https://github.com/soveran/ost

Demo

https://github.com/karmi/resque-demo

Demo

http://git.karmi.cz/resque_job_polling_demo.git

2 „Messaging”


MESSAGING

„Object Oriented Programming”

I'm sorry that I long ago coined the term "objects" for this
topic because it gets many people to focus on the lesser idea.
e big idea is "messaging" (...).
e key in making great and growable systems is much more
to design how its modules communicate rather than what
their internal properties and behaviors should be.

— Alan Kay, prototypes vs classes was: Re: Sun's HotSpot

http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html The Code of Forking Paths

MESSAGING

Three Overlooked Features of A Software System

Maintainability (changing features)

Extensibility (adding or removing features)
Testability (validating features)


MESSAGING

Asynchronous Task Processing in A Real World

When you place your order the cashier marks a coﬀee cup
with your order and places it into the queue. e queue is
quite literally a queue of coﬀee cups lined up on top of the
espresso machine. is queue decouples cashier and barista
and allows the cashier to keep taking orders even if the
barista is backed up for a moment.

— Gregor Hohpe, Starbucks Does Not Use Two-Phase Commit

http://www.eaipatterns.com/ramblings/18_starbucks.html The Code of Forking Paths

MESSAGING

AMQP

Alvaro Videla and Jason J.W. Williams, RabbitMQ in Action
p. 16


AMQP

Exchanges, Queues and Bindings

Alvaro Videla and Jason J.W. Williams, RabbitMQ in Action
p. 30


AMQP

Publishers and Consumers

1 Publisher
exchange = MQ.topic('log')
exchange.publish( {:level => level,
:time => Time.now, Message
:body => message}.to_json,

:key => "log.#{level}" ) Routing Key

2 Consumer (all logs)
MQ.queue("all logs").bind( MQ.topic('log'), :key => "log.*" ).subscribe do |header, message|
...
end

3 Consumer (error logs)
MQ.queue("error logs").bind( MQ.topic('log'), :key => "log.error" ).subscribe do |message|
...
end


ASYNCHRONOUS WORLD

Evented Programming

$.get('/data', function(data) {
$('.result').html(data);
alert('Loaded data from the server.');
});

var mydata = $.get('/data');


Ry Dahl, Node.js (2009)
http://nodejs.org

http://s3.amazonaws.com/four.livejournal/20091117/jsconf.pdf

module BalancingProxy
# ...
module Callbacks

def on_select
lambda do |backend|
puts "Selected backend: #{backend.url}"
backend.increment_counter if Backend.strategy == :balanced
end
end

def on_connect
lambda do |backend| module Server
puts "Connected"
# ...
end
end Backend.select do |backend|
conn.server backend, :host => backend.host, :port => backend.port
def on_data
lambda do |data| conn.on_connect &Callbacks.on_connect
puts "Receiving data"
conn.on_data &Callbacks.on_data
data
end conn.on_response &Callbacks.on_response
end conn.on_finish &Callbacks.on_finish
end
def on_response end
lambda do |backend, resp|
puts "Handling response"
resp
end
end

def on_finish
lambda do |backend|
puts "Finished"
backend.decrement_counter if Backend.strategy == :balanced
end
end

end

end

https://github.com/igrigorik/em-proxy/blob/master/examples/balancing.rb The Code of Forking Paths


Resume

➡ Long-running Tasks
➡ Task Queues (Redis, Resque)
➡ Maintainability, Extensibility, Testability
➡ “Monadic” Architecture (of loosely coupled parts)
➡ Messaging: AMQP, Publisher/Consumer, Routing



Watch Online

Watch this lecture online (in Czech):
http://multimedia.vse.cz/media/Viewer/?peid=51c06c512f4645289c4e9c749dc85acc1d


The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP)

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (12)

More from Karel Minarik

More from Karel Minarik (20)

Recently uploaded

Recently uploaded (20)

The Code Of The Forking Paths (Asynchronous Processing with Resque and AMQP)