Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PyCon India 2012: Celery Talk


Published on

Celery for background task processing and deferred execution

Published in: Technology
  • Login to see the comments

PyCon India 2012: Celery Talk

  1. 1. Celery for backgroundtask processing anddeferred executionPiyush Kumar & Konark Modi
  2. 2. Show of Hands : How many of us• Have used or know about celery / djcelery .• Have worked with queues .
  3. 3. Setup details :System config. wed be using: Centos 6.2 : Hosted on Virtual box Python version : Python 2.6.6 MySQL Django-1.4Packages installed : 1. Virutal env : pip install virtualenv 2. RabbitMQ 3. Celery : pip install -U celery 4. Django-Celery : pip install django-celery 5. MySQLdb module for python : easy_install MySQL , yuminstall mysql-devel python-devel 6. pytz : pip install pytz
  4. 4. Tutorial sections:• Understanding the basics – Level 0• Getting started with celery – Level 1• Next Steps with Celery – Level 2• Recap of what we’ve discussed till now• Q&A
  5. 5. Understanding the basics – Level 0• Need of job queues• What is celery ?• Understanding AMQP (Advanced Message Queueing Protocol )• Choosing a broker : Why RabbitMQ ?• Moving to console : Understanding RabbitMQ better.
  6. 6. Getting started with Celery – Level 1• Celery architecture• Setting up celery : Installation of all the components• Hands-On : A simple application• How are we using Celery• Integrating Celery with a Web-Framework : Django using djcelery• Django-celery architecture• Hands-On : Web application using celery
  7. 7. Next steps with Celery – Level 2• Adding multiple worker• Retrying tasks• Routing to multiple queues• Monitoring• Security
  8. 8. Lets get started now JUnderstanding the basics – Level 0
  9. 9. Need of Job Queues:• For executing tasks asynchronusly.• Distributed tasks.• Scheduling.• Retries.• Handling errors.• Dynamic work load.• Enhancing user experience.• Some of the use cases include: • Clickstream ( No. of hotels being watched, likes etc.) • Generate graphs based on a large data set for display on a site’s home page / Dashboard every 15 minute • Admin applications like • Thumbnails generation • Re-generate static CSS files by examining when certain items in the admin have been modified • Connecting to third-party APIs • Mass e-mailers
  10. 10. What is Celery ?• Asynschronus task queue/job queue.• Uses distributed message passing.• Supports both real-time processing + schedule jobs.• Tasks can be run concurrently on a single or more worker servers taking advantage of multiprocessing.
  11. 11. AMQP (Advanced Message Queueing Protocol )• Open standard application layer protocol for message- oriented middleware• Features of AMQP • Message orientation, • Queuing, • Routing (including point-to-point and publish-and-subscribe), • Reliability • Security • Interoperable: AMQP mandates the behaviour of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems.
  12. 12. Choosing a broker : Why RabbitMQ ?
  13. 13. Moving to console : UnderstandingRabbitMQ better
  14. 14. Getting started with celery – Level 1
  15. 15. Installing Celery :• RabbitMQ ( )• pip install -U celery
  16. 16. A simple application• Create• Run celeryd and understand its various options• Configure the result backend• Fetch the result• Call the task using file, python interpreter • Delay() • Apply_async() : Expiration, countdown• Periodic tasks
  17. 17. Our use cases:• Poll approx 300+ DB queries every 15 minutes.• Custom metric collection• Inventory management system : • Asynchronus web-interface • Periodic tasks
  18. 18. Integrating celery with a web-framework : Django with djcelery• Installation• Understanding the DB schema djcelery uses to store tasks/result sets• Writing a simple web-app
  19. 19. Next steps• Adding multiple workers• Retrying tasks• Monitoring various components of celery: • Exploring various components of our celery setup. • Setting up celery-cam Monitoring RabbitMQ• Running celeryd as a daemon in production• Security : • Signals , Tracking of tasks as they transition through different states, and inspecting return values. • Message signing.
  20. 20. Recap of what we’ve discussed till now
  21. 21. Thanks : Q & A