Constant Contact SF's Road to CD
•
0 likes•835 views
Anthony Ikeda's slides from his talk at the Automated Testing SF meetup in December 2014.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Constant Contact SF's Road to CD
Similar to Constant Contact SF's Road to CD (20)
Recently uploaded
Recently uploaded (20)
Constant Contact SF's Road to CD
- 1. Presented by : Road to CD at Constant Contact Increasing Velocity 1
- 2. Agenda • Life Before Continuous Delivery • Discovering and Implementing Continuous Delivery • Refining the Flow • Working with Continuous Integration/Delivery • Evolving Continuous Delivery 2
- 3. Quick Overview of Engineering at Constant Contact 3
- 4. Engineering • About 15-18 engineering teams each with an average size of 8-12 engineers • Typical ratio of 1 Quality Engineer per 2-3 Software Engineers • Teams function as either Feature teams, Platform teams or a combination of both 4
- 5. Engineering • Distribution includes: • Waltham, MA (Head Office) • Tribeca, NY • Battery Park, NY • San Francisco, CA • Boca Raton, FL • Loveland, CO 5
- 7. Life Before CD • Weekly deploys • Build pipelines owned and operated by Engineering Services • All deploys had to go through a request and queue process • Progressing through 5 different environments • Development, Functional, Load, Stage, Production 7
- 8. Life Before CD • Engineers coding large units of work in the Development Environment • Code was then deployed to the Functional environment • Testing implemented and executed by the Quality Engineers in the Functional testing Environment • Integration testing may happen in the Functional or Load Testing Environment • Once all tested code progressed to the Stage Environment 8
- 9. 9 Backlog In Development Ready for Dev Ready for Validation In Validation Ready for Deploy Deployed
- 10. 10 Development LoadFunctional Stage Production 30% Stable 80% Stable 90% Stable 99% Stable 99% Stable
- 11. Life Before CD • Visibility on what was being deployed was very general • We knew work was going out but troubleshooting issues tended to be tricky due to large commits • Kanban SDLC • Multitasking was a big thing 11
- 12. Discovering and Implementing Continuous Delivery 12
- 13. The Big Question 13 How do we start? Where
- 14. What is everyone else doing? • We first looked at how other teams had approached CD 14
- 15. New York, NY Office • Pre-emptive CD 15
- 16. Waltham, MA Office • Integrated CD 16
- 17. We set goals! • Development teams own their applications • Build and deploy through to production • Deliver quality code minimising defects through improved test coverage • Increase velocity by 20% by the end of 2013, another 50% by the end of 2014 • Direct to production deploys by the end of 2014 17
- 18. We formed a CD Team • Main body of governance for CD • Core development and assistance of integrations • Goto team for anything CD • Consisted of: • Product Owner • Dev Manager • Architect • QE • Engineering was utilised form the team adopting CD 18
- 19. Planning • Key to success in implementing CD • Where are you deploying? • What are the steps to a successful deploy? • What are the effects of one event to another? • How will engineers engage with CD? • How will this affect existing product work? • What are the milestones you want to deliver to a team? • Comes even before thinking of tooling 19
- 20. The Approach • Convert to a single-branch model • Hard version all artefacts (produced and consumed) • Harden test coverage • Set up the skeleton build jobs • Simple Checkout -> Build -> Unit Test -> Publish -> Deploy Pipeline • Deploy to 1 environment only then backfill 20
- 21. The Approach • We didn’t want to manage multiple environments any more just 1 environment! • Implement the practice of Feature Switches • Coach the Product owners how to write User Stories 21
- 22. Working with Continuous Delivery 22
- 23. Technology Stack • Jenkins • Open Stack (New) • LDAP • Puppet • GitHub • Artifactory (New) 23
- 24. Single Branch Model 24 Release Feature 1 Feature 2
- 25. Single Branch Model 25 Release Feature 1 Feature 2 Build Validation
- 26. Single Branch Model 26 Release Feature 1 Feature 2 Pull Request Validation Build Validation
- 27. Integration Testing • Whole environment configured to execute Integration testing • Separate Jenkins Server • Separate OpenStack Group • Owned by the QE team 27
- 28. 28 Development LoadFunctional Stage Production CD Deploy Manual Deploy Manual Deploy Manual Deploy Manual Deploy
- 29. 29 Development LoadFunctional Stage Production CD Deploy Backfill Manual Deploy Manual Deploy Manual Deploy 1 OS VM
- 30. 30 Development LoadFunctional Stage Production CD Deploy Backfill Manual Deploy Manual Deploy Backfill 1 OS VM 2 OS VMs
- 31. 31 Development LoadFunctional Stage Production CD Deploy Backfill Push Button Deploy Backfill Backfill 30% Stable 80% Stable 90% Stable 99% Stable 99% Stable 1 OS VM 2 OS VMs 4 OS VMs
- 32. 32 Development LoadFunctional Stage Production CD Deploy Backfill Push Button Deploy Backfill Backfill 30% Stable 80% Stable 90% Stable 99% Stable 99% Stable 1 OS VM 2 OS VMs 4 OS VMs 6 OS VMs
- 34. Automating Servers • Ensure that domains are properly configured • Setting up and pulling down instances in a coordinated fashion • Attaching to the correct environment 34
- 35. Monitoring & Alerts • Jenkins Job health plugin • Healthcheck dashboard • Splunk Logging • Nagios 35
- 36. Results of First Deliverable • We had direct to Load deploys happening with backfill • Test coverage up • New SDLC with QEs engaging at the beginning of the flow, not waiting for engineers to complete work • New way to view the health of the system through monitoring • Base skeleton workflow that would be used to build new pipelines 36
- 37. Discoveries • There is no main entry point when starting CD • Planning is crucial • Your SDLC will most likely change • Monitoring is key to knowing your progress • Educating the users of the system is also paramount • Everyone is involved with CD 37
- 38. Discoveries • CD is evolutionary • It will become part of your projects, not a separate project • It will help your velocity just not right away • Don’t be afraid to innovate 38
- 40. SDLC • Scaled Agile Framework • Initially taken from v1.0 • More recently v3.0 • Had to change • Epics -> Features -> Stories (-> Tasks) 40
- 41. SDLC • Needed better tracking of what was happening now • Velocity increased • A new development programme had to be put together • Moved from Kanban to Scrum 41
- 42. SDLC • Introduced 3 flows in Jira: • Epic Flow • Feature Flow • Scrum/Story Flow 42
- 43. Evolving CD 43
- 44. 2015 • Examine the gaps and address them • Better integration with GitHub, Jenkins and Jira • Improve reporting and Release Notes • Try and automate the whole process from changing a Jira Ticket status to building out a whole environment 44