SlideShare a Scribd company logo
1 of 30
Download to read offline
DevOps @
DevOps @
DNC & OFA
DNC & OFA
John Schneider                Leo Zhadanovsky
@jdotp                        @leozh
Formerly Obama for America    Director of Systems Engineering
Currently DevOps / Internet   @
Jedi @ Mozilla Foundation     Democratic National Committee
DevOps @ DNC - 2009 - 2011

Year 1
● Cleanup
● Virtualization
● Documentation
● Version Control
● Making everything HA
● Replacing all our network gear
● Automation
Labs Development Team

● Created first versions of Call Tool, Voter
  Registration Site, Canvassing iPhone App
Labs Development Team

Start of DevOps Practices at the DNC
● Agile Development
● Usage of EC2
● Continuous Integration
● Deploys using Capistrano
● Scaling up using custom AMIs
● Monitoring using Nagios, Cacti, NewRelic,
  Graphite, Pingdom, StatsD
Analytics Team

●   Lessons learned from 2010 used in 2012
●   Big Data
●   Hardware vs AWS
●   Limitations of AWS at the time
●   What would we do differently now?
●   Developed software for data analysts,
    modelers and field staff
Cloud (AWS)

● Unpredictable traffic patterns: No problem!
  (We had no clue those birther mugs would
  sell so well.)

● Makes entire org far more agile

● Apples to apples testing envs are a snap

● Cloud rules changes the way we manage
  systems
Dev + Ops = <3
            The New Mentality of Devops

                 SOA / 12 Factor

● Allow for easy migration, deployment
● All components independently scaled
  without retooling or re-architecting
● Critical for building scalable, high-availability
  apps in the cloud.
● Goes well with Route53.
Dev + Ops = <3
          The New Mentality of Devops

             Agile and Continuous
● Tool agnostic (we used Pivatol, Basecamp
  Classic, Basecamp Next, spreadsheets,
  github issues, zendesk tickets, etc)

● Small and fast releases. (we pushed ~40
  new versions per day in the last weekend)

● SHIPIT! Build 'good enough' tools and
  iterate.
Dev + Ops = <3
           The New Mentality of Devops

   Instrument Everything (and trust the data)

● CPU, traffic, response time, etc

● User/System events

● Deployment-related change metrics

● Logs are data too!
Dev + Ops = <3
            The New Mentality of Devops

     Visible Ops (aka, how to turn PMs, Devs,
          Management into Ops partners)
●   They all get Nagios, NewRelic, and
    Pagerduty accounts and training
●   Activity feed of commits, deploys, alerts,
    changes.
●   Cross-pollinate team stands, retros
●   Give everyone contextual data
Who doesn't love color charts?
Load Testing with NewRelic Deploys
Dev + Ops = <3
           The New Mentality of Devops

         Fail Early, Fail Fast, Fail Well

● It is not if, but when.
● Learn how to detect & mitigate, and then
  iterate.
● Reduce the time to detect, the time to
  resolve.
● Preventing all failure is not the DevOps role,
  making failure hurt less or not hurt at all is
  DevOps role
Yo, mind if we bring the donations
db down during the VP debate?
Oh Hai East Virginia Datacenter!
Other AWS Tips
-Use EBS-backed instances, AMI-based
deploys, and provisioned IOPS.
-Use SQS.
-Multi-AZ, Multi-Region early
-Autoscale from any metric, push your own
metrics to Cloudwatch even
-Setup ifttt or zapier alerts on AWS status
pages
-Check out billing broken out by tag, and
TrustedAdvisor
S3 microsites are awesome
Other Things That Helped
New Relic (We love you New Relic)
Asgard (Thanks Netflix)
StatsD/Graphite and Cubism
Loggly
BeesWithMachineGuns
Akamai DSA and GTM
Boto
PalominoDB is awesome!
Lots of caffeine
Lastly....culture, culture, culture.
Election Day
DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)

More Related Content

What's hot

Making operations visible - devopsdays tokyo 2013
Making operations visible  - devopsdays tokyo 2013Making operations visible  - devopsdays tokyo 2013
Making operations visible - devopsdays tokyo 2013
Nick Galbreath
 
Automated Configuration & Deployment of Atlassian Applications
Automated Configuration & Deployment of Atlassian ApplicationsAutomated Configuration & Deployment of Atlassian Applications
Automated Configuration & Deployment of Atlassian Applications
colleenfry
 
JIRA Performance Testing in Pictures - Edward Bukoski Michael March
JIRA Performance Testing in Pictures - Edward Bukoski Michael MarchJIRA Performance Testing in Pictures - Edward Bukoski Michael March
JIRA Performance Testing in Pictures - Edward Bukoski Michael March
Atlassian
 
Funambol: introducing SCRUM in software product development - AgileDay Italia
Funambol: introducing SCRUM in software product development - AgileDay ItaliaFunambol: introducing SCRUM in software product development - AgileDay Italia
Funambol: introducing SCRUM in software product development - AgileDay Italia
Edoardo Schepis
 

What's hot (20)

Ensuring Performance in a Fast-Paced Environment (CMG 2014)
Ensuring Performance in a Fast-Paced Environment (CMG 2014)Ensuring Performance in a Fast-Paced Environment (CMG 2014)
Ensuring Performance in a Fast-Paced Environment (CMG 2014)
 
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as CodeConfoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
 
Introduction to the Typesafe Reactive Platform
Introduction to the Typesafe Reactive PlatformIntroduction to the Typesafe Reactive Platform
Introduction to the Typesafe Reactive Platform
 
Key alias dev standard final
Key alias   dev standard finalKey alias   dev standard final
Key alias dev standard final
 
Introduction to Puppet Enterprise
Introduction to Puppet EnterpriseIntroduction to Puppet Enterprise
Introduction to Puppet Enterprise
 
(DVO204) Monitoring Strategies: Finding Signal in the Noise
(DVO204) Monitoring Strategies: Finding Signal in the Noise(DVO204) Monitoring Strategies: Finding Signal in the Noise
(DVO204) Monitoring Strategies: Finding Signal in the Noise
 
Rapid Data Analytics @ Netflix
Rapid Data Analytics @ NetflixRapid Data Analytics @ Netflix
Rapid Data Analytics @ Netflix
 
TestIstanbul 2015
TestIstanbul 2015TestIstanbul 2015
TestIstanbul 2015
 
Enterprise PHP (PHP London Conference 2008)
Enterprise PHP (PHP London Conference 2008)Enterprise PHP (PHP London Conference 2008)
Enterprise PHP (PHP London Conference 2008)
 
Continuous database deployment
Continuous database deploymentContinuous database deployment
Continuous database deployment
 
Going faster with kanban
Going faster with kanbanGoing faster with kanban
Going faster with kanban
 
Will ServerLess kill containers and Operations
Will ServerLess kill containers and OperationsWill ServerLess kill containers and Operations
Will ServerLess kill containers and Operations
 
DevOps Enterprise Summit 2016
DevOps Enterprise Summit 2016DevOps Enterprise Summit 2016
DevOps Enterprise Summit 2016
 
Making operations visible - devopsdays tokyo 2013
Making operations visible  - devopsdays tokyo 2013Making operations visible  - devopsdays tokyo 2013
Making operations visible - devopsdays tokyo 2013
 
Automated Configuration & Deployment of Atlassian Applications
Automated Configuration & Deployment of Atlassian ApplicationsAutomated Configuration & Deployment of Atlassian Applications
Automated Configuration & Deployment of Atlassian Applications
 
JIRA Data Center Implementation at Pitney Bowes - Peter Strickland
JIRA Data Center Implementation at Pitney Bowes - Peter StricklandJIRA Data Center Implementation at Pitney Bowes - Peter Strickland
JIRA Data Center Implementation at Pitney Bowes - Peter Strickland
 
Infrastructure as Code @BuildStuff2013
Infrastructure as Code @BuildStuff2013Infrastructure as Code @BuildStuff2013
Infrastructure as Code @BuildStuff2013
 
JIRA Performance Testing in Pictures - Edward Bukoski Michael March
JIRA Performance Testing in Pictures - Edward Bukoski Michael MarchJIRA Performance Testing in Pictures - Edward Bukoski Michael March
JIRA Performance Testing in Pictures - Edward Bukoski Michael March
 
Funambol: introducing SCRUM in software product development - AgileDay Italia
Funambol: introducing SCRUM in software product development - AgileDay ItaliaFunambol: introducing SCRUM in software product development - AgileDay Italia
Funambol: introducing SCRUM in software product development - AgileDay Italia
 
SuiteWorld16: Mega Volume - How TubeMogul Leverages NetSuite
SuiteWorld16: Mega Volume - How TubeMogul Leverages NetSuiteSuiteWorld16: Mega Volume - How TubeMogul Leverages NetSuite
SuiteWorld16: Mega Volume - How TubeMogul Leverages NetSuite
 

Viewers also liked

DevOps/Flow workshop for agile india 2015
DevOps/Flow workshop for agile india 2015DevOps/Flow workshop for agile india 2015
DevOps/Flow workshop for agile india 2015
Yuval Yeret
 
Devops the Microsoft Way
Devops the Microsoft WayDevops the Microsoft Way
Devops the Microsoft Way
Patrick Chanezon
 
DevOps monitoring: Feedback loops in enterprise environments
DevOps monitoring: Feedback loops in enterprise environmentsDevOps monitoring: Feedback loops in enterprise environments
DevOps monitoring: Feedback loops in enterprise environments
Jonah Kowall
 

Viewers also liked (17)

Which watcher watches CloudWatch
Which watcher watches CloudWatch Which watcher watches CloudWatch
Which watcher watches CloudWatch
 
Customer Ops: DevOps &lt;3 customer support
Customer Ops: DevOps &lt;3 customer supportCustomer Ops: DevOps &lt;3 customer support
Customer Ops: DevOps &lt;3 customer support
 
Survey on article extraction and comment monitoring techniques
Survey on article extraction and comment monitoring techniquesSurvey on article extraction and comment monitoring techniques
Survey on article extraction and comment monitoring techniques
 
Practical Monitoring Techniques
Practical Monitoring TechniquesPractical Monitoring Techniques
Practical Monitoring Techniques
 
DevconTLV 2014 (Jan) - DIY DevOps
DevconTLV 2014 (Jan) - DIY DevOpsDevconTLV 2014 (Jan) - DIY DevOps
DevconTLV 2014 (Jan) - DIY DevOps
 
Measured availability - Sanjay Singh - DevOps Bangalore meetup March 28th 2015
Measured availability - Sanjay Singh - DevOps Bangalore meetup March 28th 2015Measured availability - Sanjay Singh - DevOps Bangalore meetup March 28th 2015
Measured availability - Sanjay Singh - DevOps Bangalore meetup March 28th 2015
 
5 Ways ITSM can Support DevOps, an ITSM Academy Webinar
5 Ways ITSM can Support DevOps, an ITSM Academy Webinar5 Ways ITSM can Support DevOps, an ITSM Academy Webinar
5 Ways ITSM can Support DevOps, an ITSM Academy Webinar
 
DevOps Roadtrip Minneapolis
DevOps Roadtrip Minneapolis DevOps Roadtrip Minneapolis
DevOps Roadtrip Minneapolis
 
Devoxx 2014 monitoring
Devoxx 2014 monitoringDevoxx 2014 monitoring
Devoxx 2014 monitoring
 
DevOps/Flow workshop for agile india 2015
DevOps/Flow workshop for agile india 2015DevOps/Flow workshop for agile india 2015
DevOps/Flow workshop for agile india 2015
 
Run IT Support the DevOps Way
Run IT Support the DevOps WayRun IT Support the DevOps Way
Run IT Support the DevOps Way
 
Jelastic - DevOps PaaS Business with Docker Support for Service Providers
Jelastic - DevOps PaaS Business with Docker Support for Service ProvidersJelastic - DevOps PaaS Business with Docker Support for Service Providers
Jelastic - DevOps PaaS Business with Docker Support for Service Providers
 
Paris Devops - Monitoring And Feature Toggle Pattern With JMX
Paris Devops - Monitoring And Feature Toggle Pattern With JMXParis Devops - Monitoring And Feature Toggle Pattern With JMX
Paris Devops - Monitoring And Feature Toggle Pattern With JMX
 
Devops the Microsoft Way
Devops the Microsoft WayDevops the Microsoft Way
Devops the Microsoft Way
 
DevOps in the Cloud with Microsoft Azure
DevOps in the Cloud with Microsoft AzureDevOps in the Cloud with Microsoft Azure
DevOps in the Cloud with Microsoft Azure
 
DevOps monitoring: Feedback loops in enterprise environments
DevOps monitoring: Feedback loops in enterprise environmentsDevOps monitoring: Feedback loops in enterprise environments
DevOps monitoring: Feedback loops in enterprise environments
 
VIZBI 2015 Tutorial: Cytoscape, IPython, Docker, and Reproducible Network Dat...
VIZBI 2015 Tutorial: Cytoscape, IPython, Docker, and Reproducible Network Dat...VIZBI 2015 Tutorial: Cytoscape, IPython, Docker, and Reproducible Network Dat...
VIZBI 2015 Tutorial: Cytoscape, IPython, Docker, and Reproducible Network Dat...
 

Similar to DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)

Aws uk ug #8 not everything that happens in vegas stay in vegas
Aws uk ug #8   not everything that happens in vegas stay in vegasAws uk ug #8   not everything that happens in vegas stay in vegas
Aws uk ug #8 not everything that happens in vegas stay in vegas
Peter Mounce
 

Similar to DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013) (20)

Thinking DevOps in the era of the Cloud - Demi Ben-Ari
Thinking DevOps in the era of the Cloud - Demi Ben-AriThinking DevOps in the era of the Cloud - Demi Ben-Ari
Thinking DevOps in the era of the Cloud - Demi Ben-Ari
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
 
Preparing for Neo - Singapore OutSystems User Group October 2022 Meetup
Preparing for Neo - Singapore OutSystems User Group October 2022 MeetupPreparing for Neo - Singapore OutSystems User Group October 2022 Meetup
Preparing for Neo - Singapore OutSystems User Group October 2022 Meetup
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
 
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
 
InterCon 2016 - SLA vs Agilidade: uso de microserviços e monitoramento de cloud
InterCon 2016 - SLA vs Agilidade: uso de microserviços e monitoramento de cloudInterCon 2016 - SLA vs Agilidade: uso de microserviços e monitoramento de cloud
InterCon 2016 - SLA vs Agilidade: uso de microserviços e monitoramento de cloud
 
Aws uk ug #8 not everything that happens in vegas stay in vegas
Aws uk ug #8   not everything that happens in vegas stay in vegasAws uk ug #8   not everything that happens in vegas stay in vegas
Aws uk ug #8 not everything that happens in vegas stay in vegas
 
Don't Fumble the Data! Integrate Database Automation into your DevOps Toolchain
Don't Fumble the Data! Integrate Database Automation into your DevOps ToolchainDon't Fumble the Data! Integrate Database Automation into your DevOps Toolchain
Don't Fumble the Data! Integrate Database Automation into your DevOps Toolchain
 
Apache Cassandra at Target - Cassandra Summit 2014
Apache Cassandra at Target - Cassandra Summit 2014Apache Cassandra at Target - Cassandra Summit 2014
Apache Cassandra at Target - Cassandra Summit 2014
 
Innovate Better Through Machine data Analytics
Innovate Better Through Machine data AnalyticsInnovate Better Through Machine data Analytics
Innovate Better Through Machine data Analytics
 
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
 
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
 
DevOps: Automate all the things
DevOps: Automate all the thingsDevOps: Automate all the things
DevOps: Automate all the things
 
What DevOps Isn't
What DevOps Isn'tWhat DevOps Isn't
What DevOps Isn't
 
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ PanoraysQuick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
 
Dev Ops without the Ops
Dev Ops without the OpsDev Ops without the Ops
Dev Ops without the Ops
 
AWS re:Invent 2016: Automating Workflows for Analytics Pipelines (DEV401)
AWS re:Invent 2016: Automating Workflows for Analytics Pipelines (DEV401)AWS re:Invent 2016: Automating Workflows for Analytics Pipelines (DEV401)
AWS re:Invent 2016: Automating Workflows for Analytics Pipelines (DEV401)
 
Continuous Deployment
Continuous DeploymentContinuous Deployment
Continuous Deployment
 
Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShiftDevops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
 
Delivering Better Software Faster (Without Breaking Everything)
Delivering Better Software Faster (Without Breaking Everything)Delivering Better Software Faster (Without Breaking Everything)
Delivering Better Software Faster (Without Breaking Everything)
 

Recently uploaded

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Recently uploaded (20)

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024Manulife - Insurer Innovation Award 2024
Manulife - Insurer Innovation Award 2024
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 

DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)

  • 1. DevOps @ DevOps @ DNC & OFA DNC & OFA
  • 2. John Schneider Leo Zhadanovsky @jdotp @leozh Formerly Obama for America Director of Systems Engineering Currently DevOps / Internet @ Jedi @ Mozilla Foundation Democratic National Committee
  • 3. DevOps @ DNC - 2009 - 2011 Year 1 ● Cleanup ● Virtualization ● Documentation ● Version Control ● Making everything HA ● Replacing all our network gear ● Automation
  • 4. Labs Development Team ● Created first versions of Call Tool, Voter Registration Site, Canvassing iPhone App
  • 5. Labs Development Team Start of DevOps Practices at the DNC ● Agile Development ● Usage of EC2 ● Continuous Integration ● Deploys using Capistrano ● Scaling up using custom AMIs ● Monitoring using Nagios, Cacti, NewRelic, Graphite, Pingdom, StatsD
  • 6.
  • 7. Analytics Team ● Lessons learned from 2010 used in 2012 ● Big Data ● Hardware vs AWS ● Limitations of AWS at the time ● What would we do differently now? ● Developed software for data analysts, modelers and field staff
  • 8. Cloud (AWS) ● Unpredictable traffic patterns: No problem! (We had no clue those birther mugs would sell so well.) ● Makes entire org far more agile ● Apples to apples testing envs are a snap ● Cloud rules changes the way we manage systems
  • 9. Dev + Ops = <3 The New Mentality of Devops SOA / 12 Factor ● Allow for easy migration, deployment ● All components independently scaled without retooling or re-architecting ● Critical for building scalable, high-availability apps in the cloud. ● Goes well with Route53.
  • 10.
  • 11. Dev + Ops = <3 The New Mentality of Devops Agile and Continuous ● Tool agnostic (we used Pivatol, Basecamp Classic, Basecamp Next, spreadsheets, github issues, zendesk tickets, etc) ● Small and fast releases. (we pushed ~40 new versions per day in the last weekend) ● SHIPIT! Build 'good enough' tools and iterate.
  • 12.
  • 13. Dev + Ops = <3 The New Mentality of Devops Instrument Everything (and trust the data) ● CPU, traffic, response time, etc ● User/System events ● Deployment-related change metrics ● Logs are data too!
  • 14. Dev + Ops = <3 The New Mentality of Devops Visible Ops (aka, how to turn PMs, Devs, Management into Ops partners) ● They all get Nagios, NewRelic, and Pagerduty accounts and training ● Activity feed of commits, deploys, alerts, changes. ● Cross-pollinate team stands, retros ● Give everyone contextual data
  • 15. Who doesn't love color charts?
  • 16.
  • 17.
  • 18.
  • 19. Load Testing with NewRelic Deploys
  • 20. Dev + Ops = <3 The New Mentality of Devops Fail Early, Fail Fast, Fail Well ● It is not if, but when. ● Learn how to detect & mitigate, and then iterate. ● Reduce the time to detect, the time to resolve. ● Preventing all failure is not the DevOps role, making failure hurt less or not hurt at all is DevOps role
  • 21. Yo, mind if we bring the donations db down during the VP debate?
  • 22. Oh Hai East Virginia Datacenter!
  • 23. Other AWS Tips -Use EBS-backed instances, AMI-based deploys, and provisioned IOPS. -Use SQS. -Multi-AZ, Multi-Region early -Autoscale from any metric, push your own metrics to Cloudwatch even -Setup ifttt or zapier alerts on AWS status pages -Check out billing broken out by tag, and TrustedAdvisor
  • 24. S3 microsites are awesome
  • 25. Other Things That Helped New Relic (We love you New Relic) Asgard (Thanks Netflix) StatsD/Graphite and Cubism Loggly BeesWithMachineGuns Akamai DSA and GTM Boto PalominoDB is awesome! Lots of caffeine Lastly....culture, culture, culture.
  • 26.
  • 27.
  • 28.