We has discussed ways to scale an enterprise store leveraging asynchronous capabilities introduced in Magento EE via AMQP and Asynchronous Operations modules. On the example of mass update in a background task, we has covered API for queuing tasks to multiple workers and ways to handle recoverable and non-recoverable errors.

1. © 2016 Magento, Inc. Page | 1

2. © 2016 Magento, Inc. Page | 2
Improving Enterprise Store
Scalability with Queues

3. © 2016 Magento, Inc. Page | 3
Message Queue Framework in Magento EE
Overview of capabilities

4. © 2016 Magento, Inc. Page | 4
What are Queue advantages?
• Execute tasks in background process
• Asynchronously, with multiple workers
• Take advantage of RabbitMQ benefits:
– High Availability
– Scalability
– Load Balancing
– Postponed execution
– Platform Agnostic
– Open Source

5. © 2016 Magento, Inc. Page | 5
What does Magento EE add to this?
• Provides support for queuing with the
simple API
• Transparent workflow
• Takes care of topology with just xml
configurations
• Great way to eliminate blocking
requests
• Fallback queue solution based solely
on MqSQL for testing and specific
applications

6. © 2016 Magento, Inc. Page | 6
Basic Concepts
• Message is an atomic unit of processing
• Message exists in queue until explicitly deleted
• Never sent more then once
• Always sent with specific topic
• Consumers get appropriate messages by
subscribing to the desired topic

7. © 2016 Magento, Inc. Page | 7
Basic Workflow

8. © 2016 Magento, Inc. Page | 8
Publisher
• Publishes message data to the specific exchange
• Data may be represented as an arbitrary array or DTO
• Configured with xml configuration
• Simple interface

9. © 2016 Magento, Inc. Page | 9
Consumer
• Consumes message from the queue
• Passes it to the configured handler class for processing
• Again, simple xml configuration

10. © 2016 Magento, Inc. Page | 10
Batch Consumer
• For the cases where lots of messages are involved
• Consumes a batch of multiple messages of configured size
• Allows handler to process whole batch in a loop

11. © 2016 Magento, Inc. Page | 11
Configuration
• Queues configuration happens in a few files inside the module’s
“etc” directory.
• “communication.xml” is responsible for aspects of the message
queue system that all communication types have in common. AMQP
and Database connections. Values may be overridden for
deployment in “env.php”.
• “queue.xml” is responsible for defining the broker that processes the
topic.
• “queue_consumer.xml” responsible for configuring a consumer,
allowing to set it’s type and handler for messages.
• “queue_publisher.xml” configures publisher connection to the
message broker.

12. © 2016 Magento, Inc. Page | 12
Are there any disadvantages?
• Concurrent execution of similar actions
may lead to locks in specific scenarios
• No guarantee on the timeframe, when
the task will be processed
• Generally, using a single worker
decreases overall performance:
– Using Multiple workers improves
performance, but only up to a certain
grade.

13. © 2016 Magento, Inc. Page | 13
Asynchronous Operations Module
Yes, it gets even better.

14. © 2016 Magento, Inc. Page | 14
Asynchronous Operations in Magento EE
• Significantly improves “Look and Feel” for the merchant
• Allows Store Admin to send entity updates to the queue for
background processing eliminating long blocking requests
• Provides a UI for tracking status of the execution and logging
• Allows to retry certain failed operations
• Built on top of the Message Queue Framework

15. © 2016 Magento, Inc. Page | 15
Asynchronous Operations

16. © 2016 Magento, Inc. Page | 16
Magento Asynchronous Operations Workflow
• Create Operation per entity to be processed
• Form bulk of operations
• Post messages to the Queue
• Handle operations and their statuses from the Queue
Operation
Operation
Operation
Queue Handler UI

17. © 2016 Magento, Inc. Page | 17
Operation
• Operation is a single unit of work to be
processed
• Framework interprets it as a simple DTO
containing:
– Operation UUID
– Name of the destination topic in Queue
– Serialized data or DTO for processing
– Operation Status and Status Message
• Recoverably Failed operations may be
retried in a separate bulk

18. © 2016 Magento, Inc. Page | 18
Operation Statuses
• Operations can be in one of the 4 possible statuses:
– Open: still in queue
– Complete: processed successfully without errors
– Failed Recoverably: based on exception type it is possible to determine
if operation is worth retrying. For example a connection, wait or lock
exceptions, which, in some cases, may happen when dealing with the
large number of similar items
– Failed Non Recoverably: for cases when operation is failed due to
reasons which require manual interaction, for example validation errors
• Operation status is accompanied by Result Message, visible to the
store admin, which may contain more detailed information on the
problem

19. © 2016 Magento, Inc. Page | 19
Bulk Management
• Bulk is an array of operations,
identified by Unique ID
• Visible as a separate entry to the
admin user who created it, along with
the count and status of operations
within it
• Every Operation is published as a
separate message under a desired
topic name
• Bulk Management provides a simple
API to publish Operations into the
Queue

20. © 2016 Magento, Inc. Page | 20
How does it look for a store admin?

21. © 2016 Magento, Inc. Page | 21
Quick Recap

22. © 2016 Magento, Inc. Page | 22
Quick recap
• Queue is good for performing long operations in background
• Great way to avoid blocking requests
• Scales well with the number of workers, however has its limits
• Simple API on Magento's side

23. © 2016 Magento, Inc. Page | 23
Example Module
• You may take a look at the example module, which using some of
the discussed features updates Catalog Product Stock Quantities
• https://tinyurl.com/mmit17-queues
• https://github.com/ishakhsuvarov/mmit17-queue-example

24. © 2016 Magento, Inc. Page | 24
Q&A
For more questions: @ishakhsuvarov
(twitter, github or @magento.com email)