Introductory talk about Scalability and Django. Main topics: - Does Django Scale? - What is Scalability? - What is Django? - How to scale Django? Talk given at Python Barcelona (25th April 2019): https://www.meetup.com/python-185/events/260642585/

1. “Does Django scale?”
An introduction to Scalability
Python Barcelona
David Arcos

2. Hi! I’m David
●
I’m a backend developer specialized in distributed systems,
scalability and security
●
>10 years building SaaS platforms in different sectors
●
CTO at Lead Ratings
– SaaS solution that uses Machine-Learning algorithms to
predict business metrics (conversion, billing, churn...)
●
Associate professor at ESADE
– Teaching “Cloud Computing”
●
PyBCN meetup assistant since 2008, organizer since 2013

3. 1) Does Django Scale?
Questions

4. https://stackoverflow.com/questions/886221/does-django-scale

5. https://docs.djangoproject.com/en/dev/faq/general/#does-django-scale

6. 1) Does Django Scale? YES
2) What is Scalability?
Questions

7. Scalability
“The potential of a system
to be enlarged to accommodate
a growing amount of work”

8. “A Word on Scalability” (2006)
“A service is said to be scalable if when
we increase the resources in a system,
it results in increased performance in a
manner proportional to resources added”
- Werner Vogels (Amazon CTO)
https://www.allthingsdistributed.com/2006/03/a_word_on_scalability.html

9. Will it scale?
An algorithm/protocol/system does scale, if it
is suitably efficient and practical when applied
to large situations. In example:
– a large input data set
– a large number of outputs or users
– a large number of participating nodes
If it fails when increasing, it doesn’t scale.

10. Performance != Scalability
●
Performance is a quality metric:
– “The time it takes to execute one request”
●
Scalability:
– The ability to maintain that performance, under
increasing load
– Or: increase performance when adding resources

11. Where is the state?
●
Scaling is much easier if:
– The app servers are stateless
– The database keeps the state
●
So:
– Vertical scalability on (stateful) db servers
– Horizontal scalability on (stateless) app servers

12. Vertical / horizontal scaling

13. Vertical scaling
●
Scale up/down: add/remove resources to a
single node
– Adding more power (faster CPU, more CPUs, more
memory, a bigger disk…) to an existing machine
●
Very common on databases (stateful systems)
●
Easier… when possible

14. Vertical scaling… but...
●
But:
– There are hardware limits
●
RDS current limit of 488 GiB of memory (db.r4.16xlarge)
– Diminishing returns
●
At some point, the level of benefits gained is less than
the amount of money invested.
– Involves downtime
●
Keep a hot spare!
●
RDS Multi A-Z has a “standby” replica with automatic
failover

15. Horizontal scaling
●
Scale out/in: add/remove nodes to a system
– Adding new computers to a cluster
●
Very common on (stateless) web servers
●
High availability
– doesn’t matter if some nodes crash
●
Cheaper (HW is cheap, VMs are cheap)

16. Horizontal scaling.. but...
●
But:
– Architecture must support it
●
Keep the state in external db!
– Management overhead
●
Automate it!
●
Sysadmin operations must be treated as software + config

17. 1) Does Django Scale? YES
2) What is Scalability?
3) What is Django?
Questions

18. “The web framework for
perfectionists with deadlines”
https://www.djangoproject.com

19. Who uses Django?
●
Instagram
●
Pinterest
●
Spotify
●
Disqus
●
Bitbucket
●
Mozilla Firefox
●
Dropbox
●
Open Stack
●
NASA
●
The Washington Post
●
National Geographic
●
The Onion
●
Eventbrite
●
Prezi
●
Reddit Gifts
●
Open Knowledge
Foundation
●
MacArthur Foundation

22. DRF: Django REST framework
https://www.django-rest-framework.org
“A powerful and flexible toolkit for building Web APIs”

23. http://www.celeryproject.org
Celery: Distributed Task Queue

24. Other Python web frameworks

25. 1) Does Django Scale? YES
2) What is Scalability?
3) What is Django?
4) How to scale Django?
Questions

26. How to Scale Django?
●
Measure first, find bottlenecks
– Rinse and repeat
●
Focus on database performance
– Indexes, queries, replicas...
●
Cache all the things!
– Excellent cache documentation
●
Run slow tasks in async workers
– (when possible)

27. “High Performance Django” book
https://highperformancedjango.com

28. 1) Does Django Scale? YES
2) What is Scalability?
3) What is Django?
4) How to scale Django?
Questions

29. Thanks for attending!
At Lead Ratings we are looking for:
– Python Developers
– Data Engineers
– Data Scientists

30. Questions?