Pony ORM is an Object-Relational Mapper implemented in Python. It uses an unusual approach for writing database queries using Python generators. Pony analyzes the abstract syntax tree of a generator and translates it to its SQL equivalent. The translation process consists of several non-trivial stages. This talk was given at EuroPython 2014 and reveals the internal details of the translation process.

1. Object-Relational Mapper
Alexey Malashkevich @ponyorm

2. What makes Pony ORM different?
Fast object-relational mapper
which uses Python generators for
writing database queries

3. A Python generator
(p.name for p in product_list if p.price > 100)

4. A Python generator vs a SQL query
SELECT p.name
FROM Products p
WHERE p.price > 100
(p.name for p in product_list if p.price > 100)

5. (p.name for p in product_list if p.price > 100)
SELECT p.name
FROM Products p
WHERE p.price > 100
A Python generator vs a SQL query

6. (p.name for p in product_list if p.price > 100)
SELECT p.name
FROM Products p
WHERE p.price > 100
A Python generator vs a SQL query

7. (p.name for p in product_list if p.price > 100)
SELECT p.name
FROM Products p
WHERE p.price > 100
A Python generator vs a SQL query

8. The same query in Pony
SELECT p.name
FROM Products p
WHERE p.price > 100
select(p.name for p in Product if p.price > 100)

9. • Pony ORM
• Django
• SQL Alchemy
Query syntax comparison

10. Pony ORM:
select(p for p in Product
if p.name.startswith('A') and p.image is None
or p.added.year < 2014)
Query syntax comparison

11. Django:
Product.objects.filter(
Q(name__startswith='A', image__isnull=True)
| Q(added__year__lt=2014))
Query syntax comparison

12. SQLAlchemy:
session.query(Product).filter(
(Product.name.startswith('A')
& (Product.image == None))
| (extract('year', Product.added) < 2014))
Query syntax comparison

13. session.query(Product).filter(
(Product.name.startswith('A') & (Product.image == None))
| (extract('year', Product.added) < 2014))
Query syntax comparison
Product.objects.filter(
Q(name__startswith='A', image__isnull=True)
| Q(added__year__lt=2014))
select(p for p in Product
if p.name.startswith('A') and p.image is None
or p.added.year < 2014)
Pony
Django
SQLAlchemy

14. Query translation
select(p for p in Product
if p.name.startswith('A') and p.image is None
or p.added.year < 2014)
• Translation from the bytecode is fast
• The bytecode translation result is cached
• The Python generator object is used as a
cache key
Python generator object

15. Building a query step by step
q = select(o for o in Order if o.customer.id == some_id)
q = q.filter(lambda o: o.state != 'DELIVERED')
q = q.filter(lambda o: len(o.items) > 2)
q = q.order_by(Order.date_created)
q = q[10:20]
SELECT "o"."id"
FROM "Order" "o"
LEFT JOIN "OrderItem" "orderitem-1"
ON "o"."id" = "orderitem-1"."order"
WHERE "o"."customer" = ?
AND "o"."state" <> 'DELIVERED'
GROUP BY "o"."id"
HAVING COUNT("orderitem-1"."ROWID") > 2
ORDER BY "o"."date_created"
LIMIT 10 OFFSET 10

16. How Pony translates generator
expressions to SQL?

17. Python generator to SQL translation
1. Decompile bytecode and restore AST
2. Translate AST to ‘abstract SQL’
3. Translate ‘abstract SQL’ to a specific SQL
dialect

18. Python generator to SQL translation
1. Decompile bytecode and restore AST
2. Translate AST to ‘abstract SQL’
3. Translate ‘abstract SQL’ to a concrete
SQL dialect

19. Bytecode decompilation
• Using the Visitor pattern
• Methods of the Visitor object correspond
the byte code commands
• Pony keeps fragments of AST at the stack
• Each method either adds a new part of AST
or combines existing parts

20. (a + b.c) in x.y
Bytecode decompilation

21. (a + b.c) in x.y
LOAD_GLOBAL a
LOAD_FAST b
LOAD_ATTR c
BINARY_ADD
LOAD_FAST x
LOAD_ATTR y
COMPARE_OP in
Bytecode decompilation

22. Bytecode decompilation
(a + b.c) in x.y
LOAD_GLOBAL a
LOAD_FAST b
LOAD_ATTR c
BINARY_ADD
LOAD_FAST x
LOAD_ATTR y
COMPARE_OP in
Stack

23. Bytecode decompilation
(a + b.c) in x.y
> LOAD_GLOBAL a
LOAD_FAST b
LOAD_ATTR c
BINARY_ADD
LOAD_FAST x
LOAD_ATTR y
COMPARE_OP in
Stack
Name('a')

24. Bytecode decompilation
(a + b.c) in x.y
LOAD_GLOBAL a
> LOAD_FAST b
LOAD_ATTR c
BINARY_ADD
LOAD_FAST x
LOAD_ATTR y
COMPARE_OP in
Stack
Name('b')
Name('a')

25. (a + b.c) in x.y
LOAD_GLOBAL a
LOAD_FAST b
> LOAD_ATTR c
BINARY_ADD
LOAD_FAST x
LOAD_ATTR y
COMPARE_OP in
Stack
Getattr(Name('b'), 'c')
Name('a')
Bytecode decompilation

26. (a + b.c) in x.y
LOAD_GLOBAL a
LOAD_FAST b
LOAD_ATTR c
> BINARY_ADD
LOAD_FAST x
LOAD_ATTR y
COMPARE_OP in
Stack
Add(Name('a'),
Getattr(Name('b'), 'c'))
Bytecode decompilation

27. Bytecode decompilation
(a + b.c) in x.y
LOAD_GLOBAL a
LOAD_FAST b
LOAD_ATTR c
BINARY_ADD
> LOAD_FAST x
LOAD_ATTR y
COMPARE_OP in
Stack
Name('x')
Add(Name('a'),
Getattr(Name('b'), 'c'))

28. Bytecode decompilation
(a + b.c) in x.y
LOAD_GLOBAL a
LOAD_FAST b
LOAD_ATTR c
BINARY_ADD
LOAD_FAST x
> LOAD_ATTR y
COMPARE_OP in
Stack
Getattr(Name('x'), 'y')
Add(Name('a'),
Getattr(Name('b'), 'c'))

29. Bytecode decompilation
(a + b.c) in x.y
LOAD_GLOBAL a
LOAD_FAST b
LOAD_ATTR c
BINARY_ADD
LOAD_FAST x
LOAD_ATTR y
> COMPARE_OP in
Stack
Compare('in',
Add(…), Getattr(…))

30. Abstract Syntax Tree (AST)
a
in
+
.c
b
.y
x
(a + b.c) in x.y

31. Python generator to SQL translation
1. Decompile bytecode and restore AST
2. Translate AST to ‘abstract SQL’
3. Translate ‘abstract SQL’ to a concrete
SQL dialect

32. What SQL it should be translated to?
a
in
+
.c
b
.y
x
(a + b.c) in x.y

33. It depends on variables types!
What SQL it should be translated to?
(a + b.c) in x.y

34. • If a and c are numbers, y is a collection
(? + "b"."c") IN (SELECT …)
• If a and c are strings, y is a collection
CONCAT(?, "b"."c") IN (SELECT …)
• If a, c and y are strings
“x"."y" LIKE CONCAT('%', ?, "b"."c", '%')
What SQL it should be translated to?
(a + b.c) in x.y

35. • The result of translation depends on types
• If the translator analyzes node types by
itself, the logic becomes too complex
• Pony uses Monads to keep it simple
(a + b.c) in x.y
AST to SQL Translation

36. • Encapsulates the node translation logic
• Generates the result of translation - ‘the
abstract SQL’
• Can combine itself with other monads
The translator delegates the logic of translation
to monads
A Monad

37. • StringAttrMonad
• StringParamMonad
• StringExprMonad
• StringConstMonad
• DatetimeAttrMonad
• DatetimeParamMonad
• ObjectAttrMonad
• CmpMonad
• etc…
Each monad defines a set of allowed operations and can
translate itself into a part of resulting SQL query
Monad types

38. AST Translation
• Using the Visitor pattern
• Walk the tree in depth-first order
• Create monads when leaving each node

39. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b

40. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b

41. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad

42. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad

43. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad

44. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad

45. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad

46. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad

47. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad

48. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad
StringExpr
Monad

49. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad
StringExpr
Monad

50. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad
StringExpr
Monad

51. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad
StringExpr
Monad
ObjectIter
Monad

52. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad
StringExpr
Monad
ObjectIter
Monad

53. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad
StringExpr
Monad
ObjectIter
Monad
StringAttr
Monad

54. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad
StringExpr
Monad
ObjectIter
Monad
StringAttr
Monad

55. (a + b.c) in x.y
AST to SQL Translation
in
.y
x
+
a .c
b
StringParam
Monad
ObjectIter
Monad
StringAttr
Monad
StringExpr
Monad
ObjectIter
Monad
StringAttr
Monad
Cmp
Monad

56. Abstract SQL
(a + b.c) in x.y
['LIKE', ['COLUMN', 't1', 'y'],
['CONCAT',
['VALUE', '%'], ['PARAM', 'p1'],
['COLUMN', 't2', 'c'], ['VALUE', '%']
]
]
Allows to put aside the SQL dialect differences

57. Python generator to SQL translation
1. Decompile bytecode and restore AST
2. Translate AST to ‘abstract SQL’
3. Translate ‘abstract SQL’ to a specific SQL
dialect

58. Specific SQL dialects
['LIKE', ['COLUMN', 't1', 'y'],
['CONCAT',
['VALUE', '%'], ['PARAM', 'p1'],
['COLUMN', 't2', 'c'], ['VALUE', '%']
]
]
MySQL:
`t1`.`y` LIKE CONCAT('%', ?, `t2`.`c`, '%')
SQLite:
"t1"."y" LIKE '%' || ? || "t2"."c" || '%'

59. Other Pony ORM features
• Identity Map
• Automatic query optimization
• N+1 Query Problem solution
• Optimistic transactions
• Online ER Diagram Editor

60. Django ORM
s1 = Student.objects.get(pk=123)
print s1.name, s1.group.id
s2 = Student.objects.get(pk=456)
print s2.name, s2.group.id
• How many SQL queries will be executed?
• How many objects will be created?

61. Django ORM
s1 = Student.objects.get(pk=123)
print s1.name, s1.group.id
s2 = Student.objects.get(pk=456)
print s2.name, s2.group.id
Student 123

62. Django ORM
s1 = Student.objects.get(pk=123)
print s1.name, s1.group.id
s2 = Student.objects.get(pk=456)
print s2.name, s2.group.id
Student 123 Group 1

63. Django ORM
s1 = Student.objects.get(pk=123)
print s1.name, s1.group.id
s2 = Student.objects.get(pk=456)
print s2.name, s2.group.id
Student 123
Student 456
Group 1

64. Django ORM
s1 = Student.objects.get(pk=123)
print s1.name, s1.group.id
s2 = Student.objects.get(pk=456)
print s2.name, s2.group.id
Student 123
Student 456
Group 1
Group 1

65. Pony ORM
s1 = Student[123]
print s1.name, s1.group.id
s2 = Student[456]
print s2.name, s2.group.id

66. Pony ORM – seeds, IdentityMap
s1 = Student[123]
print s1.name, s1.group.id
s2 = Student[456]
print s2.name, s2.group.id
Student 123
Group 1

67. Pony ORM – seeds, IdentityMap
s1 = Student[123]
print s1.name, s1.group.id
s2 = Student[456]
print s2.name, s2.group.id
Student 123
Group 1
seed

68. Pony ORM – seeds, IdentityMap
s1 = Student[123]
print s1.name, s1.group.id
s2 = Student[456]
print s2.name, s2.group.id
Student 123
Group 1
seed

69. Pony ORM – seeds, IdentityMap
s1 = Student[123]
print s1.name, s1.group.id
s2 = Student[456]
print s2.name, s2.group.id
Student 123
Student 456
Group 1
seed

70. Pony ORM – seeds, IdentityMap
s1 = Student[123]
print s1.name, s1.group.id
s2 = Student[456]
print s2.name, s2.group.id
Student 123
Student 456
Group 1
seed

71. Solution for the N+1 Query Problem
orders = select(o for o in Order if o.total_price > 1000)
.order_by(desc(Order.id)).page(1, pagesize=5)
for o in orders:
print o.total_price, o.customer.name
1
SELECT o.id, o.total_price, o.customer_id,...
FROM "Order" o
WHERE o.total_price > 1000
ORDER BY o.id DESC
LIMIT 5

72. Order 1
Order 3
Order 4
Order 7
Order 9
Customer 1
Customer 4
Customer 7
Solution for the N+1 Query Problem

73. Order 1
Order 3
Order 4
Order 7
Order 9
Customer 1
Customer 4
Customer 7
Solution for the N+1 Query Problem
One SQL query

74. Solution for the N+1 Query Problem
1
1
SELECT c.id, c.name, …
FROM “Customer” c
WHERE c.id IN (?, ?, ?)
orders = select(o for o in Order if o.total_price > 1000)
.order_by(desc(Order.id)).page(1, pagesize=5)
for o in orders:
print o.total_price, o.customer.name
SELECT o.id, o.total_price, o.customer_id,...
FROM "Order" o
WHERE o.total_price > 1000
ORDER BY o.id DESC
LIMIT 5

75. Automatic query optimization
select(c for c in Customer
if sum(c.orders.total_price) > 1000)
SELECT "c"."id", "c"."email", "c"."password", "c"."name",
"c"."country", "c"."address"
FROM "Customer" "c"
WHERE (
SELECT coalesce(SUM("order-1"."total_price"), 0)
FROM "Order" "order-1"
WHERE "c"."id" = "order-1"."customer"
) > 1000
SELECT "c"."id"
FROM "Customer" "c"
LEFT JOIN "Order" "order-1"
ON "c"."id" = "order-1"."customer"
GROUP BY "c"."id"
HAVING coalesce(SUM("order-1"."total_price"), 0) > 1000

76. Transactions
def transfer_money(id1, id2, amount):
account1 = Account.objects.get(pk=id1)
if account1.amount < amount:
raise ValueError('Not enough funds!')
account2 = Account.object.get(pk=id2)
account1.amount -= amount
account1.save()
account2.amount += amount
account2.save()
Django ORM

77. @transaction.atomic
def transfer_money(id1, id2, amount):
account1 = Account.objects.get(pk=id1)
if account1.amount < amount:
raise ValueError('Not enough funds!')
account2 = Account.object.get(pk=id2)
account1.amount -= amount
account1.save()
account2.amount += amount
account2.save()
Transactions
Django ORM

78. @transaction.atomic
def transfer_money(id1, id2, amount):
account1 = Account.objects
.select_for_update.get(pk=id1)
if account1.amount < amount:
raise ValueError('Not enough funds!')
account2 = Account.objects
.select_for_update.get(pk=id2)
account1.amount -= amount
account1.save()
account2.amount += amount
account2.save()
Transactions
Django ORM

79. @db_session
def transfer_money(id1, id2, amount):
account1 = Account[id1]
if account1.amount < amount:
raise ValueError('Not enough funds!')
account1.amount -= amount
Account[id2].amount += amount
Transactions
Pony ORM

80. db_session
• Pony tracks which objects where changed
• No need to call save()
• Pony saves all updated objects in a single
transaction automatically on leaving the
db_session scope
Transactions

81. UPDATE Account
SET amount = :new_value
WHERE id = :id
AND amount = :old_value
Optimistic Locking

82. Optimistic Locking
• Pony tracks attributes which were read and
updated
• If object wasn’t locked using the for_update
method, Pony uses the optimistic locking
automatically

83. Entity-Relationship Diagram Editor
https://editor.ponyorm.com

84. Entity-Relationship Diagram Editor
https://editor.ponyorm.com

85. Entity-Relationship Diagram Editor
https://editor.ponyorm.com

86. Main Pony ORM features:
• Using generators for database queries
• Identity Map
• Solution for N+1 Query Problem
• Automatic query optimization
• Optimistic transactions
• Online ER-diagram editor
Wrapping up

87. • Python 3
• Microsoft SQL Server support
• Improved documentation
• Migrations
• Ansync queries
Pony roadmap

88. • Site ponyorm.com
• Twitter @ponyorm
• Github github.com/ponyorm/pony
• ER-Diagram editor editor.ponyorm.com
• Installation: pip install pony
Thank you!
Pony ORM