SlideShare a Scribd company logo
1 of 41
Download to read offline
DISQUS
                         Python at 400 500 million visitors

                         Jason Yan                               David Cramer
                         @jasonyan                                  @zeeg




                                     Got feedback? Use hashtag #sckrw



Sunday, March 13, 2011
Agenda




                •    What is DISQUS?
                •    An Overview of the Infrastructure
                •    Iterative Development and Deployment
                •    Why We Love Python




Sunday, March 13, 2011
What is DISQUS?

                                      dis·cuss • dĭ-skŭs'



                          We are a comment system with an
                         emphasis on connecting communities




                                    http://disqus.com/about/

Sunday, March 13, 2011
Embeddable Comments




Sunday, March 13, 2011
A Brief History




Sunday, March 13, 2011
Startup-ish




                •    Founded just about 4 years ago
                •    16 employees, 8 engineers
                •    Tra c increasing 15-20% a month
                •    Flat organizational structure, every
                     engineer is a product manager
                •    Fast turnaround, new feature launches
                     every week (sometimes daily)



Sunday, March 13, 2011
Tra      c

                                Number of Visitors
               500M



               375M



               250M



               125M



                   0M


                         March 2008 through March 2011


Sunday, March 13, 2011
DjangoCon 2010



                •    17,000 requests/
                     second peak
                •    450,000 websites
                •    15 million profiles
                •    75 million
                     comments
                •    250 million visitors



Sunday, March 13, 2011
Six Months Later



                •    17,000 requests/       •   25,000 requests/
                     second peak                second peak
                •    450,000 websites       •   700,000 websites
                •    15 million profiles     •   30 million profiles
                •    75 million             •   170 million
                     comments                   comments
                •    250 million visitors   •   500 million visitors



Sunday, March 13, 2011
Six Months Later




                •    September 2010: 250 million uniques
                •    March 2011: 500 million uniques


                •    Handling over 2x the tra c




Sunday, March 13, 2011
Six Months Later




                •    September 2010: ~100 servers
                •    March 2011: ~100 servers


                •    Scale diagonally




Sunday, March 13, 2011
Scaling Diagonally


                •    We still rent hardware, so there is no
                     “commodity hardware”
                     •   Cheaper to upgrade
                •    Everything is redundant
                •    Partition data where you need to, scale
                     partitions vertically
                •    Upgrade hardware (more RAM, more
                     drives, more cores)
                     •   Python apps tend to be CPU bound


Sunday, March 13, 2011
Infrastructure


                •    35% Web Servers
                     (Apache + mod_wsgi)


                •    15% Utility Servers
                     (Python scripts, background workers)


                •    20% Databases
                     (PostgreSQL, Redis, Membase)


                •    20% Load Balancing / High Availability
                     (HAProxy + Heartbeat)


                •    10% Caching servers
                     (Memcached, Varnish)




                •    Half of our servers run Python

Sunday, March 13, 2011
Python Web Servers

                •    Use what you’re comfortable with
                •    Apache + mod_wsgi vs nginx + uWSGI

                         Min         Avg     Max                Memory

                                                   60.0
         mod_wsgi                                  45.0
                                                   30.0
              uWSGI
                                                   15.0
                         0     200     400   600      0
                                                          mod_wsgi   uWSGI
                                req/sec


                •    Bottleneck is in the application

Sunday, March 13, 2011
Background Workers




                •    Lots of tasks that don’t need to be done in
                     web application process:
                     •   Crawling URLs
                     •   Updating avatars
                     •   Email notifications
                     •   Analytics
                     •   Counters



Sunday, March 13, 2011
Background Workers (cont’d)




                •    Most jobs are I/O bound
                     •   Slow external calls
                         •   Twitter is slow
                         •   Facebook is slow
                •    Could parallelize with multiple processes,
                     but...




Sunday, March 13, 2011
Background Workers (cont’d)




                •    Waste of memory
                •    Use non-blocking I/O
                     •   Celery 2.2 adds support for gevent/
                         eventlet




Sunday, March 13, 2011
Monitoring




                •    Application side: Graphite
                     •   Real-time(ish) graphing
                     •   Django front-end, Python backend
                •    Etsy’s StatsD proxy to Graphite
                     •   UDP (fire and forget)
                     •   Batches updates




Sunday, March 13, 2011
Monitoring

                •    Track application metrics
                     •   Errors, exceptions
                     •   New comments, users, sites, etc.
                     •   Anything




Sunday, March 13, 2011
Monitoring




                •    Check out Etsy’s posts:
                     •   Measure Anything, Measure Everything
                         http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/


                     •   Tracking Every Release
                         http://codeascraft.etsy.com/2010/12/08/track-every-release/




Sunday, March 13, 2011
What about the code?




Sunday, March 13, 2011
Powered By Django




Sunday, March 13, 2011
Which means...



                •    Largest Django-powered web application
                •    We fork, and even sometimes monkey
                     patch to make it scale to our needs
                     •   Fortunately, we don’t have to do too
                         much (Yay, Django!)
                     •   Unfortunately, we can’t use the whole of
                         the Django internal components (and if
                         we do, we do it in atypical ways)



Sunday, March 13, 2011
Iterative Development
                            Release Early Release Often




Sunday, March 13, 2011
Iterating Quickly

                •    Abstracting our application environment
                     •   Less dependancies locally
                     •   Rely on CI for dependency coverage
                •    Heavy use of open source packages
                     •   No NIH syndrome
                •    Deploy frequently, 3-7 times a day
                •    Lots of branches, but master is “stable”
                •    Realtime reporting on exceptions, metrics
                •    Our test suite is the main blocker (slow)

Sunday, March 13, 2011
Dealing with Deploys




Sunday, March 13, 2011
Gargoyle

                         Deploy features to portions of a user base at a
                          time to ensure smooth, measurable releases




                          Being users of our product, we actively use
                         early versions of features before public release

Sunday, March 13, 2011
The Deployment Problem




                •    Make some changes locally
                •    Run a subset of the test suite
                •    Push your commits
                •    CI server begins running tests
                •    ....




Sunday, March 13, 2011
Waiting on the test suite...




Sunday, March 13, 2011
Rinse and Repeat




                •    30 minutes later tests fail, start over
                •    Finally, deploy to a subset of servers
                     •   Open Sentry (our exception logger)
                     •   Monitor Graphite
                •    Deploy to 35 servers (~8 minutes)
                     •   Full rollback in < 30 seconds




Sunday, March 13, 2011
Wait, Sentry?




Sunday, March 13, 2011
Testing




Sunday, March 13, 2011
Testing Code


                •    Test suite takes around 25 minutes usually
                •    “Stuck” with Hudson (or Jenkins)
                     •   Most tightly integrated plugins are
                         geared towards Java developers
                •    Which framework do we use?
                     •   unittest(2), nose, doctests, LETTUCE?
                     •   We use unittest and nose
                •    Need to report code coverage, speed of
                     tests, pylint (or pyflakes)


Sunday, March 13, 2011
We Love Python




Sunday, March 13, 2011
Love-ish



                •    Many of us started with PHP or Rails
                •    Clean syntax, clear standards
                     •   All languages need PEP8.py and
                         PyFlakes
                •    Interpreted, fast... enough
                •    Very easy to learn
                     •   We all started by learning Django first,
                         then Python



Sunday, March 13, 2011
Haters Gonna Hate
                         If you could choose one thing in
                                Python to hate on...




Sunday, March 13, 2011
Better package management




Sunday, March 13, 2011
What can we do?




                •    Too many forks, too many frameworks
                     •   We need less clones, and more combined
                         e ort
                •    Improving existing Python solutions
                •    More Python solutions for existing
                     products




Sunday, March 13, 2011
Python Rocks!




Sunday, March 13, 2011
DISQUS
                           Questions?




                           psst, we’re hiring
                          jobs@disqus.com

Sunday, March 13, 2011
References




                •    Sentry (our exception tracking tool)
                     http://github.com/dcramer/django-sentry
                •    Gargoyle (feature switches)
                     https://github.com/disqus/gargoyle
                •    Django DB Utils (collection of db helpers for Django)
                     https://github.com/disqus/django-db-utils



                •    Jenkins CI
                     http://jenkins-ci.org/




                                              code.disqus.com
Sunday, March 13, 2011

More Related Content

What's hot

Hibernate in XPages
Hibernate in XPagesHibernate in XPages
Hibernate in XPagesToby Samples
 
Couchdbkit djangocong-20100425
Couchdbkit djangocong-20100425Couchdbkit djangocong-20100425
Couchdbkit djangocong-20100425guest4f2eea
 
Java for XPages Development
Java for XPages DevelopmentJava for XPages Development
Java for XPages DevelopmentTeamstudio
 
Intro to HTML5 Game Programming
Intro to HTML5 Game ProgrammingIntro to HTML5 Game Programming
Intro to HTML5 Game ProgrammingJames Williams
 
Eureka Moment UKLUG
Eureka Moment UKLUGEureka Moment UKLUG
Eureka Moment UKLUGPaul Withers
 
Ehcache 3: JSR-107 on steroids at Devoxx Morocco
Ehcache 3: JSR-107 on steroids at Devoxx MoroccoEhcache 3: JSR-107 on steroids at Devoxx Morocco
Ehcache 3: JSR-107 on steroids at Devoxx MoroccoLouis Jacomet
 
XPages and jQuery DataTables: Simplifying View Creation while Maximizing Func...
XPages and jQuery DataTables: Simplifying View Creation while Maximizing Func...XPages and jQuery DataTables: Simplifying View Creation while Maximizing Func...
XPages and jQuery DataTables: Simplifying View Creation while Maximizing Func...Teamstudio
 
2.28.17 Introducing DSpace 7 Webinar Slides
2.28.17 Introducing DSpace 7 Webinar Slides2.28.17 Introducing DSpace 7 Webinar Slides
2.28.17 Introducing DSpace 7 Webinar SlidesDuraSpace
 
Introducing CrossWorlds for IBM Domino
Introducing CrossWorlds for IBM DominoIntroducing CrossWorlds for IBM Domino
Introducing CrossWorlds for IBM DominoDaniele Vistalli
 
Styles of Applicaton Integration Using Spring
Styles of Applicaton Integration Using SpringStyles of Applicaton Integration Using Spring
Styles of Applicaton Integration Using SpringBruce Snyder
 
3D in the Browser via WebGL: It's Go Time
3D in the Browser via WebGL: It's Go Time 3D in the Browser via WebGL: It's Go Time
3D in the Browser via WebGL: It's Go Time Pascal Rettig
 
Real World Experience: Integrating DB2 with XPages
Real World Experience: Integrating DB2 with XPagesReal World Experience: Integrating DB2 with XPages
Real World Experience: Integrating DB2 with XPagesSteve_Zavocki
 
Lessons from a Dying CMS
Lessons from a Dying CMSLessons from a Dying CMS
Lessons from a Dying CMSSandy Smith
 
The State of Front End Web Development 2011
The State of Front End Web Development 2011The State of Front End Web Development 2011
The State of Front End Web Development 2011Pascal Rettig
 

What's hot (20)

Node.js
Node.jsNode.js
Node.js
 
Hibernate in XPages
Hibernate in XPagesHibernate in XPages
Hibernate in XPages
 
Wt unit 1 ppts web development process
Wt unit 1 ppts web development processWt unit 1 ppts web development process
Wt unit 1 ppts web development process
 
Wt unit 3 server side technology
Wt unit 3 server side technologyWt unit 3 server side technology
Wt unit 3 server side technology
 
Couchdbkit djangocong-20100425
Couchdbkit djangocong-20100425Couchdbkit djangocong-20100425
Couchdbkit djangocong-20100425
 
Java for XPages Development
Java for XPages DevelopmentJava for XPages Development
Java for XPages Development
 
Intro to HTML5 Game Programming
Intro to HTML5 Game ProgrammingIntro to HTML5 Game Programming
Intro to HTML5 Game Programming
 
Eureka Moment UKLUG
Eureka Moment UKLUGEureka Moment UKLUG
Eureka Moment UKLUG
 
Ehcache 3: JSR-107 on steroids at Devoxx Morocco
Ehcache 3: JSR-107 on steroids at Devoxx MoroccoEhcache 3: JSR-107 on steroids at Devoxx Morocco
Ehcache 3: JSR-107 on steroids at Devoxx Morocco
 
XPages and jQuery DataTables: Simplifying View Creation while Maximizing Func...
XPages and jQuery DataTables: Simplifying View Creation while Maximizing Func...XPages and jQuery DataTables: Simplifying View Creation while Maximizing Func...
XPages and jQuery DataTables: Simplifying View Creation while Maximizing Func...
 
2.28.17 Introducing DSpace 7 Webinar Slides
2.28.17 Introducing DSpace 7 Webinar Slides2.28.17 Introducing DSpace 7 Webinar Slides
2.28.17 Introducing DSpace 7 Webinar Slides
 
Introducing CrossWorlds for IBM Domino
Introducing CrossWorlds for IBM DominoIntroducing CrossWorlds for IBM Domino
Introducing CrossWorlds for IBM Domino
 
Styles of Applicaton Integration Using Spring
Styles of Applicaton Integration Using SpringStyles of Applicaton Integration Using Spring
Styles of Applicaton Integration Using Spring
 
Mobile HTML5
Mobile HTML5Mobile HTML5
Mobile HTML5
 
No sql findings
No sql findingsNo sql findings
No sql findings
 
3D in the Browser via WebGL: It's Go Time
3D in the Browser via WebGL: It's Go Time 3D in the Browser via WebGL: It's Go Time
3D in the Browser via WebGL: It's Go Time
 
Real World Experience: Integrating DB2 with XPages
Real World Experience: Integrating DB2 with XPagesReal World Experience: Integrating DB2 with XPages
Real World Experience: Integrating DB2 with XPages
 
Lessons from a Dying CMS
Lessons from a Dying CMSLessons from a Dying CMS
Lessons from a Dying CMS
 
Joomla Day DK 2012
Joomla Day DK 2012Joomla Day DK 2012
Joomla Day DK 2012
 
The State of Front End Web Development 2011
The State of Front End Web Development 2011The State of Front End Web Development 2011
The State of Front End Web Development 2011
 

Viewers also liked

Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)zeeg
 
Advancing Reinaldo Gonsalves’ Model of Global Economic Insertion
Advancing Reinaldo Gonsalves’ Model of Global Economic InsertionAdvancing Reinaldo Gonsalves’ Model of Global Economic Insertion
Advancing Reinaldo Gonsalves’ Model of Global Economic InsertionIan Walcott-Skinner
 
Javaday 2010: Facebook Java Framework
Javaday 2010: Facebook Java FrameworkJavaday 2010: Facebook Java Framework
Javaday 2010: Facebook Java FrameworkMatteo Baccan
 
Pondres Social Marketing event 26 oktober
Pondres Social Marketing event 26 oktoberPondres Social Marketing event 26 oktober
Pondres Social Marketing event 26 oktoberSjef Kerkhofs
 
Social media school 2011 webversie
Social media school 2011 webversieSocial media school 2011 webversie
Social media school 2011 webversieSjef Kerkhofs
 
Lecture somerset webversie
Lecture somerset webversieLecture somerset webversie
Lecture somerset webversieSjef Kerkhofs
 
Kim Brown, Joint Head of Learning Development & Diversity at London Boroughs ...
Kim Brown, Joint Head of Learning Development & Diversity at London Boroughs ...Kim Brown, Joint Head of Learning Development & Diversity at London Boroughs ...
Kim Brown, Joint Head of Learning Development & Diversity at London Boroughs ...Paul McElvaney
 
Scenario exercise 2014 mid atlantic technical communication conference DDeBoard
Scenario exercise 2014 mid atlantic technical communication conference DDeBoardScenario exercise 2014 mid atlantic technical communication conference DDeBoard
Scenario exercise 2014 mid atlantic technical communication conference DDeBoardddeboard
 
Coral Exodo - 10.06.07
Coral Exodo - 10.06.07Coral Exodo - 10.06.07
Coral Exodo - 10.06.07Jubrac Jacui
 
CasóRio Tati - 23.06.07
CasóRio Tati - 23.06.07CasóRio Tati - 23.06.07
CasóRio Tati - 23.06.07Jubrac Jacui
 
Totara Seminar: Paul Brooks, Northumberland
Totara Seminar: Paul Brooks, Northumberland Totara Seminar: Paul Brooks, Northumberland
Totara Seminar: Paul Brooks, Northumberland Paul McElvaney
 
The shifting art of animal breeding
The shifting art of animal breedingThe shifting art of animal breeding
The shifting art of animal breedingSijne Van der Beek
 
Dagaz Solutions Company Presentation
Dagaz Solutions Company PresentationDagaz Solutions Company Presentation
Dagaz Solutions Company PresentationDoug de Urioste
 
GDP-ivi9 What's new in the open source demo platform for automotive
GDP-ivi9 What's new in the open source demo platform for automotiveGDP-ivi9 What's new in the open source demo platform for automotive
GDP-ivi9 What's new in the open source demo platform for automotiveAgustin Benito Bethencourt
 
Social Media Advice for SHRM chapters
Social Media Advice for SHRM chaptersSocial Media Advice for SHRM chapters
Social Media Advice for SHRM chaptersMichael VanDervort
 

Viewers also liked (20)

Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)
 
Advancing Reinaldo Gonsalves’ Model of Global Economic Insertion
Advancing Reinaldo Gonsalves’ Model of Global Economic InsertionAdvancing Reinaldo Gonsalves’ Model of Global Economic Insertion
Advancing Reinaldo Gonsalves’ Model of Global Economic Insertion
 
Javaday 2010: Facebook Java Framework
Javaday 2010: Facebook Java FrameworkJavaday 2010: Facebook Java Framework
Javaday 2010: Facebook Java Framework
 
Pondres Social Marketing event 26 oktober
Pondres Social Marketing event 26 oktoberPondres Social Marketing event 26 oktober
Pondres Social Marketing event 26 oktober
 
Social media school 2011 webversie
Social media school 2011 webversieSocial media school 2011 webversie
Social media school 2011 webversie
 
Lecture somerset webversie
Lecture somerset webversieLecture somerset webversie
Lecture somerset webversie
 
Kim Brown, Joint Head of Learning Development & Diversity at London Boroughs ...
Kim Brown, Joint Head of Learning Development & Diversity at London Boroughs ...Kim Brown, Joint Head of Learning Development & Diversity at London Boroughs ...
Kim Brown, Joint Head of Learning Development & Diversity at London Boroughs ...
 
Scenario exercise 2014 mid atlantic technical communication conference DDeBoard
Scenario exercise 2014 mid atlantic technical communication conference DDeBoardScenario exercise 2014 mid atlantic technical communication conference DDeBoard
Scenario exercise 2014 mid atlantic technical communication conference DDeBoard
 
Coral Exodo - 10.06.07
Coral Exodo - 10.06.07Coral Exodo - 10.06.07
Coral Exodo - 10.06.07
 
Lecture 2
Lecture 2Lecture 2
Lecture 2
 
CasóRio Tati - 23.06.07
CasóRio Tati - 23.06.07CasóRio Tati - 23.06.07
CasóRio Tati - 23.06.07
 
Totara Seminar: Paul Brooks, Northumberland
Totara Seminar: Paul Brooks, Northumberland Totara Seminar: Paul Brooks, Northumberland
Totara Seminar: Paul Brooks, Northumberland
 
Podiumkunst
PodiumkunstPodiumkunst
Podiumkunst
 
Divosa v1.3
Divosa v1.3Divosa v1.3
Divosa v1.3
 
The shifting art of animal breeding
The shifting art of animal breedingThe shifting art of animal breeding
The shifting art of animal breeding
 
Dagaz Solutions Company Presentation
Dagaz Solutions Company PresentationDagaz Solutions Company Presentation
Dagaz Solutions Company Presentation
 
GDP-ivi9 What's new in the open source demo platform for automotive
GDP-ivi9 What's new in the open source demo platform for automotiveGDP-ivi9 What's new in the open source demo platform for automotive
GDP-ivi9 What's new in the open source demo platform for automotive
 
Social Media Advice for SHRM chapters
Social Media Advice for SHRM chaptersSocial Media Advice for SHRM chapters
Social Media Advice for SHRM chapters
 
刚好
刚好刚好
刚好
 
Generation Y Technology/Internet Habits
Generation Y Technology/Internet HabitsGeneration Y Technology/Internet Habits
Generation Y Technology/Internet Habits
 

Similar to PyCon 2011 Scaling Disqus

The Reluctant SysAdmin : 360|iDev Austin 2010
The Reluctant SysAdmin : 360|iDev Austin 2010The Reluctant SysAdmin : 360|iDev Austin 2010
The Reluctant SysAdmin : 360|iDev Austin 2010Voxilate
 
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Goikailan
 
Gaelyk - SpringOne2GX - 2010 - Guillaume Laforge
Gaelyk - SpringOne2GX - 2010 - Guillaume LaforgeGaelyk - SpringOne2GX - 2010 - Guillaume Laforge
Gaelyk - SpringOne2GX - 2010 - Guillaume LaforgeGuillaume Laforge
 
PyData Texas 2015 Keynote
PyData Texas 2015 KeynotePyData Texas 2015 Keynote
PyData Texas 2015 KeynotePeter Wang
 
Scaling Pinterest
Scaling PinterestScaling Pinterest
Scaling PinterestC4Media
 
Apache Geode - The First Six Months
Apache Geode -  The First Six MonthsApache Geode -  The First Six Months
Apache Geode - The First Six MonthsAnthony Baker
 
Building Scalable Systems: an asynchronous approach
Building Scalable Systems: an asynchronous approachBuilding Scalable Systems: an asynchronous approach
Building Scalable Systems: an asynchronous approachTheo Schlossnagle
 
Data Viz Barcamp, Amsterdam
Data Viz Barcamp, AmsterdamData Viz Barcamp, Amsterdam
Data Viz Barcamp, AmsterdamDan Brickley
 
Web Motion: Motion Detection on the Web
Web Motion: Motion Detection on the WebWeb Motion: Motion Detection on the Web
Web Motion: Motion Detection on the Webfisherwebdev
 
Webinar Mobile ECM Apps with Nuxeo EP
Webinar Mobile ECM Apps with Nuxeo EPWebinar Mobile ECM Apps with Nuxeo EP
Webinar Mobile ECM Apps with Nuxeo EPNuxeo
 
GT Logiciel Libre - Convention Systematic 2011
GT Logiciel Libre - Convention Systematic 2011GT Logiciel Libre - Convention Systematic 2011
GT Logiciel Libre - Convention Systematic 2011Stefane Fermigier
 
Lecture the dynamic web (2013)
Lecture   the dynamic web (2013)Lecture   the dynamic web (2013)
Lecture the dynamic web (2013)Dave Wallace
 
How Criteo Scaled and Supported Massive Growth with MongoDB (2013)
How Criteo Scaled and Supported  Massive Growth with MongoDB (2013)How Criteo Scaled and Supported  Massive Growth with MongoDB (2013)
How Criteo Scaled and Supported Massive Growth with MongoDB (2013)Julien SIMON
 
Monitoring is easy, why are we so bad at it presentation
Monitoring is easy, why are we so bad at it  presentationMonitoring is easy, why are we so bad at it  presentation
Monitoring is easy, why are we so bad at it presentationTheo Schlossnagle
 
JavaSE - The road forward
JavaSE - The road forwardJavaSE - The road forward
JavaSE - The road forwardeug3n_cojocaru
 
Power to the People: Manipulating SharePoint with Client-Side JavaScript
Power to the People:  Manipulating SharePoint with Client-Side JavaScriptPower to the People:  Manipulating SharePoint with Client-Side JavaScript
Power to the People: Manipulating SharePoint with Client-Side JavaScriptPeterBrunone
 

Similar to PyCon 2011 Scaling Disqus (20)

The Reluctant SysAdmin : 360|iDev Austin 2010
The Reluctant SysAdmin : 360|iDev Austin 2010The Reluctant SysAdmin : 360|iDev Austin 2010
The Reluctant SysAdmin : 360|iDev Austin 2010
 
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
2011 June - Singapore GTUG presentation. App Engine program update + intro to Go
 
Gaelyk - SpringOne2GX - 2010 - Guillaume Laforge
Gaelyk - SpringOne2GX - 2010 - Guillaume LaforgeGaelyk - SpringOne2GX - 2010 - Guillaume Laforge
Gaelyk - SpringOne2GX - 2010 - Guillaume Laforge
 
PyData Texas 2015 Keynote
PyData Texas 2015 KeynotePyData Texas 2015 Keynote
PyData Texas 2015 Keynote
 
Scaling Pinterest
Scaling PinterestScaling Pinterest
Scaling Pinterest
 
Apache Geode - The First Six Months
Apache Geode -  The First Six MonthsApache Geode -  The First Six Months
Apache Geode - The First Six Months
 
Building Scalable Systems: an asynchronous approach
Building Scalable Systems: an asynchronous approachBuilding Scalable Systems: an asynchronous approach
Building Scalable Systems: an asynchronous approach
 
Data Viz Barcamp, Amsterdam
Data Viz Barcamp, AmsterdamData Viz Barcamp, Amsterdam
Data Viz Barcamp, Amsterdam
 
Web Motion: Motion Detection on the Web
Web Motion: Motion Detection on the WebWeb Motion: Motion Detection on the Web
Web Motion: Motion Detection on the Web
 
Webinar Mobile ECM Apps with Nuxeo EP
Webinar Mobile ECM Apps with Nuxeo EPWebinar Mobile ECM Apps with Nuxeo EP
Webinar Mobile ECM Apps with Nuxeo EP
 
library h3lp
library h3lplibrary h3lp
library h3lp
 
App Engine Meetup
App Engine MeetupApp Engine Meetup
App Engine Meetup
 
GT Logiciel Libre - Convention Systematic 2011
GT Logiciel Libre - Convention Systematic 2011GT Logiciel Libre - Convention Systematic 2011
GT Logiciel Libre - Convention Systematic 2011
 
Lecture the dynamic web (2013)
Lecture   the dynamic web (2013)Lecture   the dynamic web (2013)
Lecture the dynamic web (2013)
 
How Criteo Scaled and Supported Massive Growth with MongoDB (2013)
How Criteo Scaled and Supported  Massive Growth with MongoDB (2013)How Criteo Scaled and Supported  Massive Growth with MongoDB (2013)
How Criteo Scaled and Supported Massive Growth with MongoDB (2013)
 
Monitoring is easy, why are we so bad at it presentation
Monitoring is easy, why are we so bad at it  presentationMonitoring is easy, why are we so bad at it  presentation
Monitoring is easy, why are we so bad at it presentation
 
Agile framework Support
Agile framework SupportAgile framework Support
Agile framework Support
 
JavaSE - The road forward
JavaSE - The road forwardJavaSE - The road forward
JavaSE - The road forward
 
Power to the People: Manipulating SharePoint with Client-Side JavaScript
Power to the People:  Manipulating SharePoint with Client-Side JavaScriptPower to the People:  Manipulating SharePoint with Client-Side JavaScript
Power to the People: Manipulating SharePoint with Client-Side JavaScript
 
nejmbi
nejmbinejmbi
nejmbi
 

More from zeeg

Tools for Development and Debugging in Python
Tools for Development and Debugging in PythonTools for Development and Debugging in Python
Tools for Development and Debugging in Pythonzeeg
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deploymentzeeg
 
Building Scalable Web Apps
Building Scalable Web AppsBuilding Scalable Web Apps
Building Scalable Web Appszeeg
 
Sentry (SF Python, Feb)
Sentry (SF Python, Feb)Sentry (SF Python, Feb)
Sentry (SF Python, Feb)zeeg
 
DjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling DisqusDjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling Disquszeeg
 
Db tips & tricks django meetup
Db tips & tricks django meetupDb tips & tricks django meetup
Db tips & tricks django meetupzeeg
 

More from zeeg (6)

Tools for Development and Debugging in Python
Tools for Development and Debugging in PythonTools for Development and Debugging in Python
Tools for Development and Debugging in Python
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deployment
 
Building Scalable Web Apps
Building Scalable Web AppsBuilding Scalable Web Apps
Building Scalable Web Apps
 
Sentry (SF Python, Feb)
Sentry (SF Python, Feb)Sentry (SF Python, Feb)
Sentry (SF Python, Feb)
 
DjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling DisqusDjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling Disqus
 
Db tips & tricks django meetup
Db tips & tricks django meetupDb tips & tricks django meetup
Db tips & tricks django meetup
 

Recently uploaded

UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 

Recently uploaded (20)

UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 

PyCon 2011 Scaling Disqus

  • 1. DISQUS Python at 400 500 million visitors Jason Yan David Cramer @jasonyan @zeeg Got feedback? Use hashtag #sckrw Sunday, March 13, 2011
  • 2. Agenda • What is DISQUS? • An Overview of the Infrastructure • Iterative Development and Deployment • Why We Love Python Sunday, March 13, 2011
  • 3. What is DISQUS? dis·cuss • dĭ-skŭs' We are a comment system with an emphasis on connecting communities http://disqus.com/about/ Sunday, March 13, 2011
  • 5. A Brief History Sunday, March 13, 2011
  • 6. Startup-ish • Founded just about 4 years ago • 16 employees, 8 engineers • Tra c increasing 15-20% a month • Flat organizational structure, every engineer is a product manager • Fast turnaround, new feature launches every week (sometimes daily) Sunday, March 13, 2011
  • 7. Tra c Number of Visitors 500M 375M 250M 125M 0M March 2008 through March 2011 Sunday, March 13, 2011
  • 8. DjangoCon 2010 • 17,000 requests/ second peak • 450,000 websites • 15 million profiles • 75 million comments • 250 million visitors Sunday, March 13, 2011
  • 9. Six Months Later • 17,000 requests/ • 25,000 requests/ second peak second peak • 450,000 websites • 700,000 websites • 15 million profiles • 30 million profiles • 75 million • 170 million comments comments • 250 million visitors • 500 million visitors Sunday, March 13, 2011
  • 10. Six Months Later • September 2010: 250 million uniques • March 2011: 500 million uniques • Handling over 2x the tra c Sunday, March 13, 2011
  • 11. Six Months Later • September 2010: ~100 servers • March 2011: ~100 servers • Scale diagonally Sunday, March 13, 2011
  • 12. Scaling Diagonally • We still rent hardware, so there is no “commodity hardware” • Cheaper to upgrade • Everything is redundant • Partition data where you need to, scale partitions vertically • Upgrade hardware (more RAM, more drives, more cores) • Python apps tend to be CPU bound Sunday, March 13, 2011
  • 13. Infrastructure • 35% Web Servers (Apache + mod_wsgi) • 15% Utility Servers (Python scripts, background workers) • 20% Databases (PostgreSQL, Redis, Membase) • 20% Load Balancing / High Availability (HAProxy + Heartbeat) • 10% Caching servers (Memcached, Varnish) • Half of our servers run Python Sunday, March 13, 2011
  • 14. Python Web Servers • Use what you’re comfortable with • Apache + mod_wsgi vs nginx + uWSGI Min Avg Max Memory 60.0 mod_wsgi 45.0 30.0 uWSGI 15.0 0 200 400 600 0 mod_wsgi uWSGI req/sec • Bottleneck is in the application Sunday, March 13, 2011
  • 15. Background Workers • Lots of tasks that don’t need to be done in web application process: • Crawling URLs • Updating avatars • Email notifications • Analytics • Counters Sunday, March 13, 2011
  • 16. Background Workers (cont’d) • Most jobs are I/O bound • Slow external calls • Twitter is slow • Facebook is slow • Could parallelize with multiple processes, but... Sunday, March 13, 2011
  • 17. Background Workers (cont’d) • Waste of memory • Use non-blocking I/O • Celery 2.2 adds support for gevent/ eventlet Sunday, March 13, 2011
  • 18. Monitoring • Application side: Graphite • Real-time(ish) graphing • Django front-end, Python backend • Etsy’s StatsD proxy to Graphite • UDP (fire and forget) • Batches updates Sunday, March 13, 2011
  • 19. Monitoring • Track application metrics • Errors, exceptions • New comments, users, sites, etc. • Anything Sunday, March 13, 2011
  • 20. Monitoring • Check out Etsy’s posts: • Measure Anything, Measure Everything http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/ • Tracking Every Release http://codeascraft.etsy.com/2010/12/08/track-every-release/ Sunday, March 13, 2011
  • 21. What about the code? Sunday, March 13, 2011
  • 22. Powered By Django Sunday, March 13, 2011
  • 23. Which means... • Largest Django-powered web application • We fork, and even sometimes monkey patch to make it scale to our needs • Fortunately, we don’t have to do too much (Yay, Django!) • Unfortunately, we can’t use the whole of the Django internal components (and if we do, we do it in atypical ways) Sunday, March 13, 2011
  • 24. Iterative Development Release Early Release Often Sunday, March 13, 2011
  • 25. Iterating Quickly • Abstracting our application environment • Less dependancies locally • Rely on CI for dependency coverage • Heavy use of open source packages • No NIH syndrome • Deploy frequently, 3-7 times a day • Lots of branches, but master is “stable” • Realtime reporting on exceptions, metrics • Our test suite is the main blocker (slow) Sunday, March 13, 2011
  • 27. Gargoyle Deploy features to portions of a user base at a time to ensure smooth, measurable releases Being users of our product, we actively use early versions of features before public release Sunday, March 13, 2011
  • 28. The Deployment Problem • Make some changes locally • Run a subset of the test suite • Push your commits • CI server begins running tests • .... Sunday, March 13, 2011
  • 29. Waiting on the test suite... Sunday, March 13, 2011
  • 30. Rinse and Repeat • 30 minutes later tests fail, start over • Finally, deploy to a subset of servers • Open Sentry (our exception logger) • Monitor Graphite • Deploy to 35 servers (~8 minutes) • Full rollback in < 30 seconds Sunday, March 13, 2011
  • 33. Testing Code • Test suite takes around 25 minutes usually • “Stuck” with Hudson (or Jenkins) • Most tightly integrated plugins are geared towards Java developers • Which framework do we use? • unittest(2), nose, doctests, LETTUCE? • We use unittest and nose • Need to report code coverage, speed of tests, pylint (or pyflakes) Sunday, March 13, 2011
  • 34. We Love Python Sunday, March 13, 2011
  • 35. Love-ish • Many of us started with PHP or Rails • Clean syntax, clear standards • All languages need PEP8.py and PyFlakes • Interpreted, fast... enough • Very easy to learn • We all started by learning Django first, then Python Sunday, March 13, 2011
  • 36. Haters Gonna Hate If you could choose one thing in Python to hate on... Sunday, March 13, 2011
  • 38. What can we do? • Too many forks, too many frameworks • We need less clones, and more combined e ort • Improving existing Python solutions • More Python solutions for existing products Sunday, March 13, 2011
  • 40. DISQUS Questions? psst, we’re hiring jobs@disqus.com Sunday, March 13, 2011
  • 41. References • Sentry (our exception tracking tool) http://github.com/dcramer/django-sentry • Gargoyle (feature switches) https://github.com/disqus/gargoyle • Django DB Utils (collection of db helpers for Django) https://github.com/disqus/django-db-utils • Jenkins CI http://jenkins-ci.org/ code.disqus.com Sunday, March 13, 2011

Editor's Notes

  1. Hi. I&apos;m Jason (and I&apos;m David), and we&apos;re from Disqus.
  2. For those of you who are not familiar with us, DISQUS is a comment systemthat focuses on connecting communities. We power discussions on such sites as CNN, IGN, andmore recently Engadget and TechCrunch. Our company was founded back in 2007 by my co-founder,Daniel Ha, and I back where we started working out of our dorm room.Our decision to use Django came down primarily to our dislike for PHP whichwe were previously using. Since then, we&apos;ve grown Disqus to over 250+million visitors a month.
  3. Show of hands, How many of you know what DISQUS is?
  4. We&apos;ve peaked at over 17,000 requests per second, to Django, and we currentlypower comments on nearly half a million websites which accounts for more than15 million profiles who have left over 75 million comments.