Using PGQ for batches reliability needs

1. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
How to abuse PGQ for batches reliability needs
Dimitri Fontaine
October 17, 2008
Dimitri Fontaine How to abuse PGQ for batches reliability needs

2. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Table of contents
1 Batches needs
2 PGQ features
3 Mix and Match
4 Conclusion
Dimitri Fontaine How to abuse PGQ for batches reliability needs

3. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Database processing oriented batches
If you’re managing an OLTP
system, you probably have out of
line processing to get done, and
probably are using cron batches
and home made daemons.
Dimitri Fontaine How to abuse PGQ for batches reliability needs

4. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Database processing oriented batches
If you’re managing an OLTP
system, you probably have out of
line processing to get done, and
probably are using cron batches
and home made daemons.
Example
while( true ) {
// what a nice daemon!
}
Dimitri Fontaine How to abuse PGQ for batches reliability needs

5. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Database processing oriented batches
If you’re managing an OLTP
system, you probably have out of Of course you want them
line processing to get done, and reliable, easy to monitor and
probably are using cron batches control (logs)
and home made daemons.
Example
while( true ) {
// what a nice daemon!
}
Dimitri Fontaine How to abuse PGQ for batches reliability needs

6. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Database processing oriented batches
If you’re managing an OLTP
system, you probably have out of Of course you want them
line processing to get done, and reliable, easy to monitor and
probably are using cron batches control (logs)
and home made daemons. out of a developer screen
Example session
while( true ) {
// what a nice daemon!
}
Dimitri Fontaine How to abuse PGQ for batches reliability needs

7. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Database processing oriented batches
If you’re managing an OLTP
system, you probably have out of Of course you want them
line processing to get done, and reliable, easy to monitor and
probably are using cron batches control (logs)
and home made daemons. out of a developer screen
Example session
while( true ) { easy to stop & restart
// what a nice daemon!
}
Dimitri Fontaine How to abuse PGQ for batches reliability needs

8. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Database processing oriented batches
If you’re managing an OLTP
system, you probably have out of Of course you want them
line processing to get done, and reliable, easy to monitor and
probably are using cron batches control (logs)
and home made daemons. out of a developer screen
Example session
while( true ) { easy to stop & restart
// what a nice daemon! to reuse existing models
}
Dimitri Fontaine How to abuse PGQ for batches reliability needs

9. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Reusing Open-Source code?
PGQ is a queuing solution implemented as a PostgreSQL extension
module, providing an SQL and a python API. It offers the producer
multi consumers subscribtion model and is the transport layer of
the londiste replication solution.
Dimitri Fontaine How to abuse PGQ for batches reliability needs

10. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Reusing Open-Source code?
PGQ is a queuing solution implemented as a PostgreSQL extension
module, providing an SQL and a python API. It offers the producer
multi consumers subscribtion model and is the transport layer of
the londiste replication solution.
PGQ is
performant (maintaining 3 tables, using TRUNCATE)
Dimitri Fontaine How to abuse PGQ for batches reliability needs

11. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Reusing Open-Source code?
PGQ is a queuing solution implemented as a PostgreSQL extension
module, providing an SQL and a python API. It offers the producer
multi consumers subscribtion model and is the transport layer of
the londiste replication solution.
PGQ is
performant (maintaining 3 tables, using TRUNCATE)
easy to install and monitor
Dimitri Fontaine How to abuse PGQ for batches reliability needs

12. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Reusing Open-Source code?
PGQ is a queuing solution implemented as a PostgreSQL extension
module, providing an SQL and a python API. It offers the producer
multi consumers subscribtion model and is the transport layer of
the londiste replication solution.
PGQ is
performant (maintaining 3 tables, using TRUNCATE)
easy to install and monitor
robust
Dimitri Fontaine How to abuse PGQ for batches reliability needs

13. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Implementing a daemon atop PGQ
Skytools comes with two middleware for you to abuse to make
daemons with, the python DBScript facility and the PHP
libphp-pgq one.
Dimitri Fontaine How to abuse PGQ for batches reliability needs

14. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Implementing a daemon atop PGQ
Skytools comes with two middleware for you to abuse to make
daemons with, the python DBScript facility and the PHP
libphp-pgq one.
For the python version see resources on the internet. Here we’re
dealing with the PHP one. PHP is inferior a language but we
sometime have to support it, to leverage existing model classes.
Dimitri Fontaine How to abuse PGQ for batches reliability needs

15. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
libphp-pgq
You subclass a PGQConsumer superclass and implement two
methods:
config
process event
Dimitri Fontaine How to abuse PGQ for batches reliability needs

16. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
libphp-pgq
You subclass a PGQConsumer superclass and implement two
methods:
config
process event
The transaction BEGIN and COMMIT are managed by the class you
inherited code from, just process data and return one of
PGQ_EVENT_OK
PGQ_EVENT_FAILED
PGQ_EVENT_RETRY
PGQ_ABORT_BATCH
Dimitri Fontaine How to abuse PGQ for batches reliability needs

17. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
libphp-pgq abstract classes
Different main classes are available for you to subclass, depending
on what you want to do. Some documentation is available online
at http://pgsql.tapoueh.org/pgq/pgq-php/
Dimitri Fontaine How to abuse PGQ for batches reliability needs

18. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
libphp-pgq abstract classes
Different main classes are available for you to subclass, depending
on what you want to do. Some documentation is available online
at http://pgsql.tapoueh.org/pgq/pgq-php/
PGQConsumer
Dimitri Fontaine How to abuse PGQ for batches reliability needs

19. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
libphp-pgq abstract classes
Different main classes are available for you to subclass, depending
on what you want to do. Some documentation is available online
at http://pgsql.tapoueh.org/pgq/pgq-php/
PGQConsumer
PGQRemoteConsumer
Dimitri Fontaine How to abuse PGQ for batches reliability needs

20. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
libphp-pgq abstract classes
Different main classes are available for you to subclass, depending
on what you want to do. Some documentation is available online
at http://pgsql.tapoueh.org/pgq/pgq-php/
PGQConsumer
PGQRemoteConsumer
PGQEventRemoteConsumer
Dimitri Fontaine How to abuse PGQ for batches reliability needs

21. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Gone live!
We have several PGQ daemons in our live environments, monitored
with nagios and munin, and we’re able to easily control them. It’s
much better than previously.
Dimitri Fontaine How to abuse PGQ for batches reliability needs

22. Outline
Batches needs
PGQ features
Mix and Match
Conclusion
Gone live!
We have several PGQ daemons in our live environments, monitored
with nagios and munin, and we’re able to easily control them. It’s
much better than previously.
Example
# mydaemon.php start
# mydaemon.php status
# mydaemon.php logmore
# mydaemon.php kill
Dimitri Fontaine How to abuse PGQ for batches reliability needs