Tutorial sections:• Understanding the basics – Level 0• Getting started with celery – Level 1• Next Steps with Celery – Level 2• Recap of what we’ve discussed till now• Q&A
Understanding the basics – Level 0• Need of job queues• What is celery ?• Understanding AMQP (Advanced Message Queueing Protocol )• Choosing a broker : Why RabbitMQ ?• Moving to console : Understanding RabbitMQ better.
Getting started with Celery – Level 1• Celery architecture• Setting up celery : Installation of all the components• Hands-On : A simple application• How are we using Celery• Integrating Celery with a Web-Framework : Django using djcelery• Django-celery architecture• Hands-On : Web application using celery
Next steps with Celery – Level 2• Adding multiple worker• Retrying tasks• Routing to multiple queues• Monitoring• Security
Lets get started now JUnderstanding the basics – Level 0
Need of Job Queues:• For executing tasks asynchronusly.• Distributed tasks.• Scheduling.• Retries.• Handling errors.• Dynamic work load.• Enhancing user experience.• Some of the use cases include: • Clickstream ( No. of hotels being watched, likes etc.) • Generate graphs based on a large data set for display on a site’s home page / Dashboard every 15 minute • Admin applications like • Thumbnails generation • Re-generate static CSS files by examining when certain items in the admin have been modified • Connecting to third-party APIs • Mass e-mailers
What is Celery ?• Asynschronus task queue/job queue.• Uses distributed message passing.• Supports both real-time processing + schedule jobs.• Tasks can be run concurrently on a single or more worker servers taking advantage of multiprocessing.
AMQP (Advanced Message Queueing Protocol )• Open standard application layer protocol for message- oriented middleware• Features of AMQP • Message orientation, • Queuing, • Routing (including point-to-point and publish-and-subscribe), • Reliability • Security • Interoperable: AMQP mandates the behaviour of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems.
A simple application• Create tasks.py• Run celeryd and understand its various options• Configure the result backend• Fetch the result• Call the task using file, python interpreter • Delay() • Apply_async() : Expiration, countdown• Periodic tasks
Our use cases:• Poll approx 300+ DB queries every 15 minutes.• Custom metric collection• Inventory management system : • Asynchronus web-interface • Periodic tasks
Integrating celery with a web-framework : Django with djcelery• Installation• Understanding the DB schema djcelery uses to store tasks/result sets• Writing a simple web-app
Next steps• Adding multiple workers• Retrying tasks• Monitoring various components of celery: • Exploring various components of our celery setup. • Setting up celery-cam Monitoring RabbitMQ• Running celeryd as a daemon in production• Security : • Signals , Tracking of tasks as they transition through different states, and inspecting return values. • Message signing.