1) DevOps practices were implemented at the Democratic National Committee between 2009-2011 to improve systems, including virtualization, documentation, version control, automation, and making everything highly available.
2) Early agile development practices and use of AWS services like EC2 and continuous integration helped scale applications for the DNC.
3) Lessons from 2010 were applied in 2012, including use of big data, evaluating hardware vs AWS costs and limitations, and developing analytics tools for various teams.
2. John Schneider Leo Zhadanovsky
@jdotp @leozh
Formerly Obama for America Director of Systems Engineering
Currently DevOps / Internet @
Jedi @ Mozilla Foundation Democratic National Committee
3. DevOps @ DNC - 2009 - 2011
Year 1
● Cleanup
● Virtualization
● Documentation
● Version Control
● Making everything HA
● Replacing all our network gear
● Automation
4. Labs Development Team
● Created first versions of Call Tool, Voter
Registration Site, Canvassing iPhone App
5. Labs Development Team
Start of DevOps Practices at the DNC
● Agile Development
● Usage of EC2
● Continuous Integration
● Deploys using Capistrano
● Scaling up using custom AMIs
● Monitoring using Nagios, Cacti, NewRelic,
Graphite, Pingdom, StatsD
6.
7. Analytics Team
● Lessons learned from 2010 used in 2012
● Big Data
● Hardware vs AWS
● Limitations of AWS at the time
● What would we do differently now?
● Developed software for data analysts,
modelers and field staff
8. Cloud (AWS)
● Unpredictable traffic patterns: No problem!
(We had no clue those birther mugs would
sell so well.)
● Makes entire org far more agile
● Apples to apples testing envs are a snap
● Cloud rules changes the way we manage
systems
9. Dev + Ops = <3
The New Mentality of Devops
SOA / 12 Factor
● Allow for easy migration, deployment
● All components independently scaled
without retooling or re-architecting
● Critical for building scalable, high-availability
apps in the cloud.
● Goes well with Route53.
10.
11. Dev + Ops = <3
The New Mentality of Devops
Agile and Continuous
● Tool agnostic (we used Pivatol, Basecamp
Classic, Basecamp Next, spreadsheets,
github issues, zendesk tickets, etc)
● Small and fast releases. (we pushed ~40
new versions per day in the last weekend)
● SHIPIT! Build 'good enough' tools and
iterate.
12.
13. Dev + Ops = <3
The New Mentality of Devops
Instrument Everything (and trust the data)
● CPU, traffic, response time, etc
● User/System events
● Deployment-related change metrics
● Logs are data too!
14. Dev + Ops = <3
The New Mentality of Devops
Visible Ops (aka, how to turn PMs, Devs,
Management into Ops partners)
● They all get Nagios, NewRelic, and
Pagerduty accounts and training
● Activity feed of commits, deploys, alerts,
changes.
● Cross-pollinate team stands, retros
● Give everyone contextual data
20. Dev + Ops = <3
The New Mentality of Devops
Fail Early, Fail Fast, Fail Well
● It is not if, but when.
● Learn how to detect & mitigate, and then
iterate.
● Reduce the time to detect, the time to
resolve.
● Preventing all failure is not the DevOps role,
making failure hurt less or not hurt at all is
DevOps role
21. Yo, mind if we bring the donations
db down during the VP debate?
23. Other AWS Tips
-Use EBS-backed instances, AMI-based
deploys, and provisioned IOPS.
-Use SQS.
-Multi-AZ, Multi-Region early
-Autoscale from any metric, push your own
metrics to Cloudwatch even
-Setup ifttt or zapier alerts on AWS status
pages
-Check out billing broken out by tag, and
TrustedAdvisor
25. Other Things That Helped
New Relic (We love you New Relic)
Asgard (Thanks Netflix)
StatsD/Graphite and Cubism
Loggly
BeesWithMachineGuns
Akamai DSA and GTM
Boto
PalominoDB is awesome!
Lots of caffeine
Lastly....culture, culture, culture.