We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.
Finally, we wrap up with a quick example of a task queue in PHP using Redis.
https://github.com/bryanhelmig/phqueue
7. def new_message(request):
user = get_user_or_404(request)
message = request.POST.get('message', None)
if not message:
raise Http404
user.save_new_message(message)
for friend in user.friends.all():
friend.send_email(message)
return redirect(reverse('dashboard'))
8. def new_message(request):
user = get_user_or_404(request)
message = request.POST.get('message', None)
if not message:
raise Http404
user.save_new_message(message)
for friend in user.friends.all():
friend.send_email(message)
return redirect(reverse('dashboard'))
9. the problem:
that works good for like 0-6 friends... but
what if you have 100,000? or more? will
your user ever get a response?
10. bad idea #1: ignore
make your users wait through your long
request/response cycle.
your users are important, right?
11. bad idea #2: ajax
return the page fast with JS code that calls
another script in the browser’s background.
duplicate calls & http cycles are not cool.
12. bad idea #3: cronjob
make a email_friends table with user_id &
message column. cron every 5/10 minutes.
backlogs will destroy you.
13. good idea: queues
the task to potentially email thousands of
users is put into a queue to be dealt with
later, leaving you to return the response.
14. @task
def alert_friends(user_id, message):
user = User.objects.get(id=user_id)
for friend in user.friends.all():
friend.send_email(message)
def new_message(request):
user = get_user_or_404(request)
message = request.POST.get('message', None)
if not message:
raise Http404
user.save_new_message(message)
alert_friends.delay(user.id, message)
return redirect(reverse('dashboard'))
15. RULE #1:
adding a task to a queue should be
faster than performing the task itself.
16. RULE #2:
you should consume tasks faster than you
produce them. if not, add more workers.
17. queue libraries:
1. celery (python) earlier...
2. resque (ruby) up next...
3. build your own grand finalé!
18. class MessageSend
def self.perform(user_id, message)
user = User.find(user_id)
user.friends.each do |friend|
friend.send_email(message)
end
end
end
class MessageController < ActionController::Base
def new_message
render(file: 'public/404.html',
status: 404) unless params[:message]
current_user.save_new_message(params[:message])
Resque.enqueue(MessageSend,
current_user.id, params[:message])
redirect_to dashboard_url
end
end
19. let’s build our own!
and let’s do it with redis, in php and
in less than 50 lines of code!
(not including the task code)
20. quick redis aside:
so, redis is a key-value store. values can
be strings, blobs, lists or hashes. we’ll use
lists and RPUSH and LPOP.
21. step #1: task runner
i’m envisioning this as a Task class with
various methods named after each task.
each method accepts an array $params.
22. class Tasks {
public function email_friends($params) {
$user_id = $params->user_id;
$message = $params->message;
# get $friends from a db query...
$friends = array('paul@example.com',
'john@example.com');
foreach ($friends as $friend) {
echo "Fake email ".$friend.
" with ".$message."n";
}
}
}
23. step #2: worker
i think an infinite loop with a blocking
redis BLPOP will work. then it needs to run
the right method on the Task object.
24. include_once 'redis.php'; # sets up $redis, $queue
include_once 'tasks.php';
$tasks = new Tasks();
while (true) {
# BLPOP will block until it gets an item..
$data = $redis->blpop($queue, 0);
$obj = json_decode($data[1]); # grab json...
$method = $obj->task_name;
$params = $obj->params;
# calls the task: Task->$task_name($params)...
call_user_func(array($tasks, $method), $params);
}
25. step #3: run it!
we will need to run the worker:
➜ php worker.php
26. step #4: producer
let’s make a handy add_task function for
easy placement of tasks into the queue.
27. include_once 'redis.php'; # sets up $redis & $queue
function add_task($task_name, $params) {
global $redis, $queue;
$data = Array('task_name' => $task_name,
'params' => $params);
$json = json_encode($data)
$redis->rpush($queue, $json);
}
# an example of our task api...
add_task('email_friends',
array('user_id' => 1234,
'message' => 'I just bought a car!')
);
28. step #5: watch it
a command line that runs a worker is
standard. use something like supervisord
or god to run it & monitor it.
29. things missing:
our php example does not: store return
results, handle errors, route tasks,
degrade gracefully, log activity, etc...