1. Anatomy of a large
Django site
Andy McKay Mozilla
mozilla

2. Vancouver
mozilla

3. Python
Zope and Plone
...now at Mozilla
mozilla

4. Using Django
http://www.djangoproject.com
Credit: http://www.flickr.com/photos/abiavati/3110357974/
mozilla

5. 1. About the site
2. Performance
3. Localisation
4. Reuse
mozilla

6. 1. About the site
mozilla

7. mozilla

8. All code is open:
https://github.com/jbalogh/zamboni
mozilla

9. All* bugs are open:
https://bugzilla.mozilla.org
mozilla

10. Convert from
CakePHP (remora)
to Django (zamboni)
mozilla

11. Credit: http://www.flickr.com/photos/improbcat/4177702580/
mozilla

12. Changing one URL at a time
from CakePHP to Django
General trend to move
away from PHP and do
more Python and Django
mozilla

13. How large?
250k+ addons
150 mn views month
500 mn API hits day
mozilla

14. Lines of code
PHP 40k
Python 18k
mozilla

15. Lines of code
PHP 40k
Python 18k
Unit tests 15k
mozilla

16. running both php and
python side by side. a
few issues on that
No pages go out until
they are faster
mozilla

17. 3 zeus load balancers
24 django (and php)
1 mysql + 4 slaves
3 memcached
3 sphinx
1 rabbitmq + 2 celeryd
1 redis master + 1 slave
Credit: http://www.flickr.com/photos/tbridge/15300843/
mozilla

18. 2. Performance
mozilla

19. As usual, database bottleneck
mozilla

20. Cache machine
http://bit.ly/cache-machine
Credit: http://www.flickr.com/photos/mwichary/4063534688/
mozilla

21. from django.db import models
import caching.base
class Addon(caching.base.CachingMixin,
models.Model):
...
status = models.IntegerField()
objects = caching.base.CachingManager()
available as a mixin
need to addin the custom
manager
mozilla

22. >>> Addon.objects.filter(status=public)
>>> len(connection.queries)
13
mozilla

23. >>> Addon.objects.filter(status=public)
>>> len(connection.queries)
13
>>> Addon.objects.filter(status=public)
>>> len(connection.queries)
13
mozilla

24. Invalidation
mozilla

25. md5(‘select... a’) [addon 3615]
mozilla

26. md5(‘select... a’) [addon 3615]
addon 3615 md5(‘select... a’)
mozilla

27. md5(‘select... a’) [addon 3615]
md5(‘select... b’) [addon 3615, addon 1685]
md5(‘select... a’)
addon 3615
md5(‘select... b’)
mozilla

28. Memcached
rules = cache.get(3615)
rules.add('select...')
cache.set(3615, rules)
mozilla

29. Redis
redis.SADD(3615, ‘select...’)
mozilla

30. Home page
20+ addons
400+ sql queries
mozilla

31. add-on version
version
version files
standard answer in
django is select-related
files
files
mozilla

32. django: select_related()
http://bit.ly/select-related
mozilla

33. simonw
Transformer
http://bit.ly/queryset-transform
mozilla

34. @staticmethod
def transformer(addons):
addon_dict = dict((a.id, a) for a in addons)
vs = filter(None, (a.current_version_id for a in addons)
versions = list(Version.objects.filter(id__in=vs))
for version in versions:
addon_dict[version.addon_id].current_version = version
slightly outdated
example
mozilla

35. big SQL statements... :(
14313 character
Home page
20+ addons
~14 sql queries
mozilla

36. Update
Called on startup
about:config extensions.update.url
mozilla

37. Incoming 8,000 req/sec
Uncached 1,600 req/sec
Im used to Plone in the
bad old days
mozilla

38. Incoming 8,000 req/sec
Uncached 1,600 req/sec
PHP 550 req/sec
Im used to Plone in the
bad old days
mozilla

39. v1
Plain
Django
PHP Python
mozilla

40. v2
Min. SQL
queries
PHP Python
mozilla

41. oh god
mozilla

42. v3
Django and
raw SQL
max-requests 200,
actually we hit 210
PHP Python
mozilla

43. v4
WSGI
no Django
PHP Python
mozilla

44. v5
Pooling,
optimised
queries
Thats 700 req/sec
PHP Python which translates into
mozilla

45. Reducing the SQL queries...
doesn’t always help
mozilla

46. mySQL query cache is fast
mozilla

47. Celery
http://celeryproject.org
Credit: http://www.flickr.com/photos/chiotsrun/3843988392/
mozilla

48. Push things async
email
image processing
add-on validation
specifically fixing data
changes bet ween php
and python
mozilla

49. from celeryutils import task
@task
def update_tag(tag, **kw):
tag.update_stat()
mozilla

50. from celeryutils import task
@task(rate_limit='60/h')
def update_tag(tag, **kw):
tag.update_stat()
mozilla

51. from tasks import update_tag
update_tag.delay(tag)
mozilla

52. Measurement
mozilla

53. Timing Middleware
http://bit.ly/timing-ware
Credit: http://www.flickr.com/photos/wwarby/3297205226/
mozilla

54. mozilla

55. 3. Localization
mozilla

56. show site in arabic?
40+ languages
including rtl
mozilla

57. content translated
and
templates
Database strings
mozilla

58. class Addon(caching.base.CachingMixin,
models.Model):
...
name = models.ForeignKey(Translation)
mozilla

59. addon.name = 'name'
addon.save()
mozilla

60. addon.name = 'name'
addon.save()
addon.name = {'fr': 'la nomme'}
addon.save()
mozilla

61. Templates
mozilla

62. Django
{% blocktrans with app=request.APP %}
Add-ons for {{ app }}
{% endblocktrans %}
mozilla

63. Jinja2
http://jinja.pocoo.org/
{{ _('Add-ons for {0}')|f(request.APP) }}
mozilla

64. Python Unicode hell
UnicodeDecodeError: 'ascii' codec can't
decode byte 0xd0 in position 16: ordinal
not in range(128)
mozilla

65. 4. Reuse
mozilla

66. Bleach
http://pypi.python.org/pypi/
Credit: http://www.flickr.com/photos/maisonbisson/3350954463/
mozilla

67. >>> bleach.clean('an
<script>evil()</script>
example')
'an &lt;script&gt;evil()&lt;/
script&gt; example'
mozilla

68. >>> bleach.linkify('an http://
ex.com url')
'an <a href="http://ex.com"
rel="nofollow">http://ex.com</
a> url'
mozilla

69. Javascript tests
mozilla

70. django-qunit
http://bit.ly/django-qunit
kumar
mozilla

71. test('English', function() {
z.refreshL10n('en-us');
equals($('textarea:visible', this.sandbox).text().trim(),
'Firebug integrates with Firefox to put ' +
'a wealth of development tools...');
});
mozilla

72. test('Japanese', function() {
z.refreshL10n('ja');
equals($('textarea:visible', this.sandbox).text().trim(),
'Firebug Web ' +
' Firefox' +
' ');
});
mozilla

73. So we use hudson for CI,
but haven’t got the
automated tests in yet
Hoping to do this via
jstestnet
Use HudsonJenkins
http://bit.ly/jstestnet
mozilla

74. pep 8
py flakes
MacCabe
Flake8
http://bit.ly/flake8
Credit: http://www.flickr.com/photos/nebarnix/357779131/
mozilla

75. ~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.py
apps/editors/tasks.py:1: 'datetime' imported but unused
apps/editors/tasks.py:3: 'stat' imported but unused
mozilla

76. Playdoh
http://bit.ly/mozilla-playdoh
fred wenzel
Credit: http://www.flickr.com/photos/ahmee/97960570/
mozilla

77. Basis for new projects
mozilla

78. Celery support
Jinja2 support
Simple migrations
By default:
SHA-512 password hashing
X-Frame-Options: DENY
secure and httponly flags on cookies
fred wenzel
mozilla

79. Take inspiration from...
but not the best for you
for example jinja2 which
makes integration with
lots of django addons
possible, but a bit harder
mozilla

80. Questions?
@andymckay
andym@mozilla.com
andym on irc.freenode.net, irc.mozilla.org
mozilla