Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps: Sprinkle Dev, Sprinkle Ops, Let's make Cake, not Mud Pies


Published on

Brian Paulsmeyer, a Sr. Architect at Centric St. Louis, spoke about DevOps on September 29th at Agile Gravy Conference in St. Louis. Here's his presentation, which starts with Agile development pitfalls that plague teams, moves into the actual capabilities that a team requires to be successful, and finally describes concrete implementations to achieve “Done Means Done” development.

Published in: Technology
  • Login to see the comments

  • Be the first to like this

DevOps: Sprinkle Dev, Sprinkle Ops, Let's make Cake, not Mud Pies

  1. 1. Sprinkle Dev, Sprinkle Ops, lets make Cake not Mud Pies Brian Paulsmeyer Agile Gravy: September 29, 2016
  2. 2. Agile Development Pitfalls … that warn us of mistakes in our agile implementation 10/4/2016 1
  3. 3. Waterfall in 2 Week Iterations •No ordering of stories •Gold plating of features •“Agile Waterfall” Assumptions • Learning from usage of system won’t change features • “All features will come together” in last day of last sprint • “Pause the world for 3+ years for us to finish” • “Value Proposition: All or Nothing” • Architecture can be “known” and not change • Deployments are still hard •Unfortunate ending for “Agile Waterfall” • “We ran out of money, and have nothing in production.” 10/4/ 2
  4. 4. Constant Redefining of “Definition of Done” •Done should simply mean done •Burndown/Burnup no longer reliable •Feature inversion occurs when time spent on defects and maintenance overtakes feature work 10/4/ 3 20 20 20 18 17 17 16 16 15 13 0 0 0 2 3 3 4 4 5 7 1 2 3 4 5 6 7 8 9 10 FEATURE INVERSION Features Defects and Hidden Work
  5. 5. Only focusing on Unit Tests and Manual Testing •Unit tests • Won’t ensure business workflows • Only allows minor refactoring • Failure: Mars Climate Orbiter (1999) • Team A: Pounds for unit of force • Team B: Newtons for unit of force (1 lb = 4.45 N) • Disintegrated in Mars atmosphere! • Each “Unit”/Component worked in isolation 10/4/ 4
  6. 6. Continuous Delivery and DevOps … simple concept, requires diligent implementation 10/4/2016 5
  7. 7. No more “Throw It Over the Wall” 10/4/ 6
  8. 8. DevOps enables Continuous Delivery •Team performs activities across disciplines •Apply principles from iteration 0 through delivery 10/4/ 7 Regression UAT Integration Testing DevOps – Full Lifecycle Delivery Pipeline and Container Management Local Environments Builds Unit Tests Merge Procedures Log Interpretation Dependency Mgmt Dev Deployments Traditional DEV activities Traditional OPS activities Traditional QA activities Prod Deployments Performance Mgmt Monitoring QA Deployments Branch Mgmt Change Mgmt Log Management Configurations Integration Deployments Performance testing
  9. 9. What It Takes Whether it be security, regulatory, software limitations, etc., there are many reasons why organizations say CD is impossible 10/4/ 8 Tooling Process Culture Architecture
  10. 10. Basic Definitions and Levels of Maturity Continuous… 10/4/ 9 Automated Deployment to ProductionDeployment Manual Compile/Build, Manual Testing, Manual DeploymentPain Automated Deployment to Integration/Staging/Pre-Prod Deployment Toolset Constant across all Environments Delivery Automated Integration/Workflow Tests Automated Component Builds Automated Component Tests* Integration Automated Unit Tests * Many teams mistakenly consider Continuous Integration (CI) complete with only automated unit tests
  11. 11. Pipeline •Recipe for building, deploying, and testing the system •Grouping of discrete stages for a complete workflow •Coordinates disparate technologies •Provides feedback of system state across environments 10/4/ 10 Business & Customer Develop Customer Deploy to Production Deploy to Staging Deploy to QA Build and Unit Test Automated Integration / Workflow Testing
  12. 12. Detailed Pipeline 10/4/ 11 Business & Customer Backlog Consolidate Features and Hotfixes Iteration Planning Build Server“X” Stories Development 1 Active Story Per Developer 1 Commit Acceptance Test Customer Build Artifact Acceptance Test Environment QA Testing QA Environment Auto-Trigger Pipeline: Installs To AIT Environment Executes Smoke and Acceptance Tests (<10 min) Manual Trigger Pipeline: Installs To QA Environment Executes Smoke Tests UAT Testing Manual Trigger Pipeline: UAT Environment UAT Smoke & UAT Tests Production Manual Trigger Pipeline: Prod Environment Prod Smoke Tests UAT Environment
  13. 13. Product Pipeline 10/4/ 12
  14. 14. Pipeline Workflow Tools … to speed implementation 10/4/2016 13
  15. 15. Pipeline Stage Implementation •Pipeline workflow steps trigger external tools 10/4/ 14 Business & Customer Develop Customer Deploy to Production Deploy to Staging Deploy to QA Build and Unit Test Java .Net TypeScript PHP etc. Ant Maven MSBuild Make etc. JUnit Nunit etc. Bash PowerShell Chef/Puppet VMs Docker/CoreOS etc. Automated Integration / Workflow Testing Selenium Cucumber JBehave SpecFlow Codeception etc. Same Tools as Deploy to QA Only Configuration Differences Same Tools as Deploy to QA Only Configuration Differences Build Tool Jobs (ex. Jenkins)
  16. 16. Key Pipeline Coordination Tool Features •Pipeline Triggers • Automatic on code check-ins to source control • Manual • Programmatic API •Workflow • Parallel steps • Sequential steps • Failure-only steps • Graphical view of steps •Artifact Tracking • Traceability from check-in to final deployment • View of deployed version in each environment • History of deployments • Ability to trigger updates to agile tracking tools (ex. JIRA) •Environment Support • Same deploy scripts, allows configuration changes per environment 10/4/ 15
  17. 17. Pipeline Workflow Tool : Jenkins •On-Site •Version 2.0 Provides Workflow View •Software Language Agnostic 10/4/ 16
  18. 18. Pipeline Workflow Tool : ThoughtWorks Go •On-Site •Software Language Agnostic 10/4/ 17
  19. 19. Pipeline Workflow Tool : Octopus Deploy •.Net 10/4/ 18
  20. 20. Automated Testing … so every day can be deployment day 10/4/2016 19
  21. 21. Automated Component/Integration/Workflow Testing •Opinions of why automated testing isn’t needed • “Manual testing is sufficient” • “Business only wants to spend money on features” • “Developers only want to code features” • “Our <insert technology> is too hard to test with automation” • “Automated tests run too slow” 10/4/ 20 Best Case: Missed Deadline Worst Case: Catastrophic Failure in Production Result of skipping automated tests
  22. 22. Selenium WebDriver Direct Test using Selenium WebDriver WebDriver driver = new FirefoxDriver(); // Navigate to webpage driver.get(""); // Find the Search Field (UI Locator) WebElement element = driver.findElement(“Search")); // Fill in the data element.sendKeys(“something interesting"); // Now submit the form. WebDriver will find the form that contains the element element.submit(); // Assert the test passed 10/4/ 21 Selenium WebDriver Web Page Test Test Test Test
  23. 23. Page Object Model 10/4/ 22 • Separate Test Code and UI Locators • Supports Don’t Repeat Yourself (DRY) for Element Locators • Less Technical Test Case SearchPage page = new SearchPage(selenium); // Enter Search information“something interesting"); // Now submit the form page.submit(); // Assert the test passed Page Object Model(s) Web Page Test Test Test Test Selenium WebDriver ServerAPI
  24. 24. BDD Testing • Gherkin (English) Text File • Step Definitions Map Between English Test and System • Business Readable Test Case Given user searches for “something interesting” When user performs the Search Then assert something is true 10/4/ 23 Page Object Model(s) Web Page Step Definition Step Definition Step Definition Selenium WebDriver ServerAPITest Test Test Test
  25. 25. Capabilities … we need to be successful 10/4/2016 24
  26. 26. Rapid Response to Changing Business •Need to be able to react to disruptive technology and business models •Stagnant architecture limits new technology adoption • Changes blocked by fear of breaking system • Inability to deploy new systems to production • Unable to adopt new security upgrades • Unable to adopt new software versions 10/4/ 25
  27. 27. Drive Defects From Most Used Features “You can’t test quality in” At a minimum make sure defects aren’t in the standard user workflows 10/4/ 26
  28. 28. Adjustable/Changeable Architecture •Magic Quadrant Architecture • “Best” choice today ≠ Best choice tomorrow • Failed technologies and companies in all parts of magic quadrant •Natural product End-of-Life requires new architecture •Continuous Delivery and DevOps allows fluid architecture changes 10/4/ 27 (Object-Oriented Database)
  29. 29. Cadence •What if everyday was a normal day •What if those normal days were release days •What if those release days were multiple releases throughout the day •Low Risk On-Demand Releases 10/4/ 28
  30. 30. Deployment Styles … for production deployment 10/4/2016 29
  31. 31. Standard Deployment •Deployment Steps • Disable user access • Upgrade servers from V1 to V2 • Test deployment • Enable user access • Upgrade destabilizes running environment •Problem Mitigation • Re-Enable backup if possible • Developer troubleshooting requires customer downtime 10/4/ 30
  32. 32. Blue Green Router / Load Balancer Blue-Green Deployment •Two Identical Production Environments • Blue • Green •One Active: Router Controlled •Deployment Steps (Green Active) • Deploy to Blue environment – No users affected • Perform smoke tests on Blue environment • Optional: Switch subset of users to Blue (similar to Canary releases) • Switch all users to Blue environment •Problem Mitigation • Don’t switch to 2nd environment • Developer troubleshooting has no customer impact 10/4/ 31
  33. 33. Canary Router / Load Balancer Canary Deployment •One Production Environment •Deployment Steps • Deploy to 1 server (Canary) Optional: Provision and build new server from scratch • Router sends portion of users to Canary • Random • Internal users only • User-type based (ex. Test users) • Gradually upgrade servers to V2 • Gradually switch users to V2 servers •Problem Mitigation • Don’t switch additional users to Canary environment • Developer troubleshooting has no customer impact 10/4/ 32
  34. 34. Case Studies … that show it can be done
  35. 35. Before Build Automation •23 Teams •  150 Team Members (Developers, Quality Engineers, Scrum Masters) •  100 Developers •Service Oriented Architecture • Managers, Engines, Data Access •Baseline End of PI6 (SAFe) • PI6 demo system setup • Two weeks to setup • Manual Installations – not repeatable • Individual Teams created “magic instructions” in isolation to install and run solution • Some teams unable to run full solution • Custom data loaders used to skip workflow steps • Developers unable to test full workflow • Movement of data through workflow steps untested till end of PI • Update of component versions caused “Day of Downtime” for developers to fix unexpected issues 10/4/ 34
  36. 36. After Build Automation: Jenkins Easy Button Builds • Starting Point: a machine with Windows — that’s it! • The rest is automated: − Provisioning: verify/install tools takes ~50 minutes − Puppet Agent, PowerShell, 7-Zip, VNC − SQL Server, .Net, Visual Studio − Noticia Repono Client − Install Custom Software takes 5-10 minutes − Clean system from any previous runs − Install/Configure Base Components − Install System − Install and Run Smoke Tests At this point the system is ready to be used to do further testing, demos, etc. • Deploy cycle repeats ChunkyMonkey Vanilla
  37. 37. Definition of Done … that makes our customers happy 10/4/2016 36
  38. 38. Continuous Delivery Definition of Done •Automated Deployments • Fully Automatic or Push-Button Deploys • Same Executable Scripts for all Environments • No Deployment Hero •Automated Testing • Unit • Integration • Full System •Always Shippable • No Temporary Code • No Untested Code • Feature Toggles •One Backlog • Production Bug Fixes • New Features 10/4/ 37
  39. 39. About Me 10/4/ 38 • Our highest priority is building lifelong relationships with clients based on trust, respect, and collaboration. • We invest in our talented team and support their well being by keeping them challenged and inspired. • Our localized company structure allows us to play very active roles in the lives of our families and community. This relationship-centric focus keeps us passionate, committed and motivated in all facets of our lives. That’s why our team is here to stay. Centric Consulting • 17 years as a software developer and architect • Full stack developer in Java, .Net, TypeScript (Angular 2), and PHP • DevOps tooling experience including VMs, Docker, Chef, and Puppet • Continuous Delivery and Automated Testing implementations • Experience across industries including FDA and SEC regulated Continue the Journey • Thank You for Attending!