SlideShare a Scribd company logo
1 of 21
Ten (Hard-Won) Lessons
of the DevOps Transition
Randy Shoup
@randyshoup
linkedin.com/in/randyshoup
1. Reorganize Teams
Around Ownership
• End-to-end Ownership
o Small, cross-functional team owns application / service from design to
deployment to retirement
o Team has inside it all skill sets needed to do the job
o Depends on other teams for supporting services
o Able to move very rapidly and independently
• “You build it, you run it”
o The same team that builds the software operates the software
o No separate maintenance or sustaining engineering team
1. Reorganize Teams
Around Ownership
• E.g., KIXEYE and MySQL
o Development team wrote the SQL, issued all the queries
o DBA / Ops team responsible for performance and uptime
o Splitting ownership between teams was counterproductive and disruptive
• Alternative strategies
o Centrally-maintained persistence service
OR
o Customer manages its own persistence
2. Lose the
Ticket Culture
Ticket Culture Ownership Culture
Do what is asked for Do what is needed
One-way communication Two-way collaboration
Goal is to close the ticket Goal is product success
Reactive approach Proactive approach
Reinforces silos Reinforces collaboration
Prioritizes process Prioritizes results
3. Replace Approvals
With Code
• Reduce or eliminate approval bodies
o E.g., eBay Architecture Review Board
o (-) Too late
o (-) Too slow
o (-) Too disengaged from details
• Package expertise in code
o Smart, experienced people build their knowledge into code
o Teams with specialized skills (databases, security, compliance, etc.) provide a
service, library, or tool
3. Replace Approvals
With Code
• E.g., Security at Google
o Provide secure foundations by maintaining lower-level libraries and services
o Provide self-service penetration tests, vulnerability assessments, etc.
The easiest way to “enforce” a
standard practice is with
working code.
4. Enforce a
Service Mentality
• Vendor-Customer Discipline
o Service team is a vendor; the products are its customers
o Service is useful only to the extent it provides value to its customers
• Customer can choose to use service or not (!)
o Customer team is responsible for deciding what is best for their use case
o Use the right tool for the right job
• Provides powerful incentives
o Service must be *strictly better* than the alternatives of build, buy, borrow
5. Charge for
Usage
• Charge customers for *usage* of the service
o Aligns economic incentives of customer and provider
o Motivates both sides to optimize efficiency
• Free usage leads to waste
o No incentive to control usage or find more efficient alternatives
• E.g., App Engine usage at Google
o Charging particularly egregious internal customer led to 10x reduction in usage
6. Prioritize
Quality
• Quality, Performance, and Reliability are “Priority-0
features”
o “Stop the line” if there is a degradation
o Equally important to users as product features or engaging user experience
• Developers write tests and code together
o Continuous testing of features, performance, load
o Confidence to make risky changes
• “Slow down to speed up”
o Catch bugs earlier, fail faster
6. Prioritize
Quality
• E.g., Development Process at Google
o Code reviews before submission
o Automated tests for everything
o Single searchable source code repository
 Internal Open Source Model
o Not “here is a bug report”
o Instead “here is the bug; here is the code fix; here is the test that verifies the fix”

7. Start Investing
in Testing
• Write functional tests around a component
o If you can only write a few tests, they should be meaningful ones
o End-to-end tests exercise more meaningful customer-visible capabilities than unit
tests
• Fail any build that breaks a test
• Keep ratcheting up the tests
o For every new feature, add tests for that feature
o For every new bug, add a test that reproduces the bug and verifies the fix
8. Actively Manage
Technical Debt
• Maintain sustainable and well-understood level of debt
o Denominated in engineering effort to fix
o Plan for how and when you will pay it off
o Track feature work vs. accrued debt over time
• “Don’t have time to do it right” ?
o WRONG  – Don’t have time to do it twice (!)
o The more constrained you are on time and resources, the more important it is to
do it solidly the first time
Vicious Cycle
of Technical Debt
Technical
Debt
“No time
to do it
right”
Quick-
and-dirty
Virtuous Cycle
of Investment
Solid
Foundation
Confidence
Faster and
Better
Invest in
Quality
9. Share
On-call Duties
• All members of the team rotate on-call responsibilities
o Strongest motivator to build in solid monitoring and diagnosis capabilities
o Best way to learn the real-world behavior of the system
o Best way to develop empathy for customers and other team members
• Train via on-call “apprenticeship”
o 1. Apprentice starts as secondary on-call, experienced engineer is primary
o 2. Apprentice is primary, experienced engineer is secondary
o 3. Apprentice graduates
10. Make Post-Mortems
Truly Blameless
• Overcoming blame culture takes work
o Institutional memory of blame is long
o E.g., Initial post-mortems at KIXEYE elicited tons of fear
• Constantly reinforce learning over blame
o When you say “blameless”, you have to really mean it (!)
o Don’t ask “what did you do?”, ask “what did you learn?”
10. Make Post-Mortems
Truly Blameless
• Open and Honest Discussion
o Document exactly what happened
o What went right
o What went wrong
• Focus on Learning and Improvement
o How should we change process, technology, documentation, etc.
o How could we have automated the problems away?
o How could we have diagnosed more quickly?
• Take fear and personalization out of it
 Engineers will compete to take personal responsibility (!)
 “Finally we can fix that broken system” 
Top Five
Takeaways
• 1. Reorganize Teams Around Ownership
• 2. Replace Approvals With Code
• 3. Prioritize Quality
• 4. Actively Manage Technical Debt
• 5. Make Post-Mortems Truly Blameless
What I Could
Use Help With
• Encouraging leaders to lose the blame culture
• Measuring productivity in a principled way
• Overcoming resistance to taking the pager
Thank You!
• @randyshoup
• linkedin.com/in/randyshoup

More Related Content

What's hot

Minimum Viable Architecture -- Good Enough is Good Enough in a Startup
Minimum Viable Architecture -- Good Enough is Good Enough in a StartupMinimum Viable Architecture -- Good Enough is Good Enough in a Startup
Minimum Viable Architecture -- Good Enough is Good Enough in a StartupRandy Shoup
 
Pragmatic Microservices
Pragmatic MicroservicesPragmatic Microservices
Pragmatic MicroservicesRandy Shoup
 
Scaling Your Architecture for the Long Term
Scaling Your Architecture for the Long TermScaling Your Architecture for the Long Term
Scaling Your Architecture for the Long TermRandy Shoup
 
Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020Randy Shoup
 
Anatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and LessonsAnatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and LessonsRandy Shoup
 
Minimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good EnoughMinimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good EnoughRandy Shoup
 
An Agile Approach to Machine Learning
An Agile Approach to Machine LearningAn Agile Approach to Machine Learning
An Agile Approach to Machine LearningRandy Shoup
 
Monoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesMonoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesRandy Shoup
 
Moving Fast at Scale
Moving Fast at ScaleMoving Fast at Scale
Moving Fast at ScaleRandy Shoup
 
The Importance of Culture: Building and Sustaining Effective Engineering Org...
The Importance of Culture:  Building and Sustaining Effective Engineering Org...The Importance of Culture:  Building and Sustaining Effective Engineering Org...
The Importance of Culture: Building and Sustaining Effective Engineering Org...Randy Shoup
 
Why Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the CloudWhy Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the CloudRandy Shoup
 
DevOpsDays Silicon Valley 2014 - The Game of Operations
DevOpsDays Silicon Valley 2014 - The Game of OperationsDevOpsDays Silicon Valley 2014 - The Game of Operations
DevOpsDays Silicon Valley 2014 - The Game of OperationsRandy Shoup
 
Managing Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and EventsManaging Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and EventsRandy Shoup
 
DevOps Picc12 Management Talk
DevOps Picc12 Management TalkDevOps Picc12 Management Talk
DevOps Picc12 Management TalkMichael Rembetsy
 
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsScaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsRandy Shoup
 
CodeOne 2018 - Better software, faster: principles of Continuous Delivery and...
CodeOne 2018 - Better software, faster: principles of Continuous Delivery and...CodeOne 2018 - Better software, faster: principles of Continuous Delivery and...
CodeOne 2018 - Better software, faster: principles of Continuous Delivery and...Bert Jan Schrijver
 
DevoxxUK 2019 - Better software, faster.
DevoxxUK 2019 - Better software, faster.DevoxxUK 2019 - Better software, faster.
DevoxxUK 2019 - Better software, faster.Bert Jan Schrijver
 
OpenValue meetup June 2019 - Better, software faster: Principles of Continuou...
OpenValue meetup June 2019 - Better, software faster: Principles of Continuou...OpenValue meetup June 2019 - Better, software faster: Principles of Continuou...
OpenValue meetup June 2019 - Better, software faster: Principles of Continuou...Bert Jan Schrijver
 
JavaZone 2019 - Better software, faster: Principles of Continuous Delivery an...
JavaZone 2019 - Better software, faster: Principles of Continuous Delivery an...JavaZone 2019 - Better software, faster: Principles of Continuous Delivery an...
JavaZone 2019 - Better software, faster: Principles of Continuous Delivery an...Bert Jan Schrijver
 
DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How? DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How? John Willis
 

What's hot (20)

Minimum Viable Architecture -- Good Enough is Good Enough in a Startup
Minimum Viable Architecture -- Good Enough is Good Enough in a StartupMinimum Viable Architecture -- Good Enough is Good Enough in a Startup
Minimum Viable Architecture -- Good Enough is Good Enough in a Startup
 
Pragmatic Microservices
Pragmatic MicroservicesPragmatic Microservices
Pragmatic Microservices
 
Scaling Your Architecture for the Long Term
Scaling Your Architecture for the Long TermScaling Your Architecture for the Long Term
Scaling Your Architecture for the Long Term
 
Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020Minimal Viable Architecture - Silicon Slopes 2020
Minimal Viable Architecture - Silicon Slopes 2020
 
Anatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and LessonsAnatomy of Three Incidents -- Commonalities and Lessons
Anatomy of Three Incidents -- Commonalities and Lessons
 
Minimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good EnoughMinimum Viable Architecture - Good Enough is Good Enough
Minimum Viable Architecture - Good Enough is Good Enough
 
An Agile Approach to Machine Learning
An Agile Approach to Machine LearningAn Agile Approach to Machine Learning
An Agile Approach to Machine Learning
 
Monoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesMonoliths, Migrations, and Microservices
Monoliths, Migrations, and Microservices
 
Moving Fast at Scale
Moving Fast at ScaleMoving Fast at Scale
Moving Fast at Scale
 
The Importance of Culture: Building and Sustaining Effective Engineering Org...
The Importance of Culture:  Building and Sustaining Effective Engineering Org...The Importance of Culture:  Building and Sustaining Effective Engineering Org...
The Importance of Culture: Building and Sustaining Effective Engineering Org...
 
Why Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the CloudWhy Enterprises Are Embracing the Cloud
Why Enterprises Are Embracing the Cloud
 
DevOpsDays Silicon Valley 2014 - The Game of Operations
DevOpsDays Silicon Valley 2014 - The Game of OperationsDevOpsDays Silicon Valley 2014 - The Game of Operations
DevOpsDays Silicon Valley 2014 - The Game of Operations
 
Managing Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and EventsManaging Data at Scale - Microservices and Events
Managing Data at Scale - Microservices and Events
 
DevOps Picc12 Management Talk
DevOps Picc12 Management TalkDevOps Picc12 Management Talk
DevOps Picc12 Management Talk
 
Scaling Your Architecture with Services and Events
Scaling Your Architecture with Services and EventsScaling Your Architecture with Services and Events
Scaling Your Architecture with Services and Events
 
CodeOne 2018 - Better software, faster: principles of Continuous Delivery and...
CodeOne 2018 - Better software, faster: principles of Continuous Delivery and...CodeOne 2018 - Better software, faster: principles of Continuous Delivery and...
CodeOne 2018 - Better software, faster: principles of Continuous Delivery and...
 
DevoxxUK 2019 - Better software, faster.
DevoxxUK 2019 - Better software, faster.DevoxxUK 2019 - Better software, faster.
DevoxxUK 2019 - Better software, faster.
 
OpenValue meetup June 2019 - Better, software faster: Principles of Continuou...
OpenValue meetup June 2019 - Better, software faster: Principles of Continuou...OpenValue meetup June 2019 - Better, software faster: Principles of Continuou...
OpenValue meetup June 2019 - Better, software faster: Principles of Continuou...
 
JavaZone 2019 - Better software, faster: Principles of Continuous Delivery an...
JavaZone 2019 - Better software, faster: Principles of Continuous Delivery an...JavaZone 2019 - Better software, faster: Principles of Continuous Delivery an...
JavaZone 2019 - Better software, faster: Principles of Continuous Delivery an...
 
DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How? DOES16 London - Better Faster Cheaper .. How?
DOES16 London - Better Faster Cheaper .. How?
 

Similar to Ten Lessons of the DevOps Transition

Agile Testing
Agile Testing Agile Testing
Agile Testing VersionOne
 
Session 1 - The Agile vs Non agile divide.pptx
Session 1 - The Agile vs Non agile divide.pptxSession 1 - The Agile vs Non agile divide.pptx
Session 1 - The Agile vs Non agile divide.pptxWatchDogs6
 
Beyond "Quality Assurance"
Beyond "Quality Assurance"Beyond "Quality Assurance"
Beyond "Quality Assurance"Jason Benton
 
Agile testingandautomation
Agile testingandautomationAgile testingandautomation
Agile testingandautomationjeisner
 
The Lean Enterprise
The Lean EnterpriseThe Lean Enterprise
The Lean EnterpriseRyan Dorrell
 
DevOps culture: Computer scientists are only human ... ;)
DevOps culture: Computer scientists are only human ... ;)DevOps culture: Computer scientists are only human ... ;)
DevOps culture: Computer scientists are only human ... ;)Jörg Hastreiter
 
WhoIsKenDeLong
WhoIsKenDeLongWhoIsKenDeLong
WhoIsKenDeLongKen DeLong
 
Technical Writing for Consultants
Technical Writing for ConsultantsTechnical Writing for Consultants
Technical Writing for ConsultantsDilum Bandara
 
Better User Onboarding (for web & mobile products)
Better User Onboarding (for web & mobile products)Better User Onboarding (for web & mobile products)
Better User Onboarding (for web & mobile products)Pulkit Agrawal
 
Adopting Agile
Adopting AgileAdopting Agile
Adopting AgileCoverity
 
ADDO19 - Automate or not from the beginning that is the question
ADDO19 - Automate or not from the beginning that is the questionADDO19 - Automate or not from the beginning that is the question
ADDO19 - Automate or not from the beginning that is the questionEnrique Carbonell
 
DevOps in action - Azure DevOps
DevOps in action - Azure DevOpsDevOps in action - Azure DevOps
DevOps in action - Azure DevOpsXPDays
 
When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"
When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"
When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"admford
 
Ericriesleanstartuppresentationforweb2
Ericriesleanstartuppresentationforweb2Ericriesleanstartuppresentationforweb2
Ericriesleanstartuppresentationforweb2Edmund FOng
 
Offshore IT Projects - Best Practices
Offshore IT Projects - Best PracticesOffshore IT Projects - Best Practices
Offshore IT Projects - Best PracticesVasantha Gullapalli
 

Similar to Ten Lessons of the DevOps Transition (20)

Agile Testing
Agile Testing Agile Testing
Agile Testing
 
Software testing
Software testingSoftware testing
Software testing
 
The Divide.pptx
The Divide.pptxThe Divide.pptx
The Divide.pptx
 
Session 1 - The Agile vs Non agile divide.pptx
Session 1 - The Agile vs Non agile divide.pptxSession 1 - The Agile vs Non agile divide.pptx
Session 1 - The Agile vs Non agile divide.pptx
 
Lean Startup 301
Lean Startup 301Lean Startup 301
Lean Startup 301
 
Beyond "Quality Assurance"
Beyond "Quality Assurance"Beyond "Quality Assurance"
Beyond "Quality Assurance"
 
Agile testingandautomation
Agile testingandautomationAgile testingandautomation
Agile testingandautomation
 
The Lean Enterprise
The Lean EnterpriseThe Lean Enterprise
The Lean Enterprise
 
DevOps culture: Computer scientists are only human ... ;)
DevOps culture: Computer scientists are only human ... ;)DevOps culture: Computer scientists are only human ... ;)
DevOps culture: Computer scientists are only human ... ;)
 
WhoIsKenDeLong
WhoIsKenDeLongWhoIsKenDeLong
WhoIsKenDeLong
 
Technical Writing for Consultants
Technical Writing for ConsultantsTechnical Writing for Consultants
Technical Writing for Consultants
 
Better User Onboarding (for web & mobile products)
Better User Onboarding (for web & mobile products)Better User Onboarding (for web & mobile products)
Better User Onboarding (for web & mobile products)
 
Adopting Agile
Adopting AgileAdopting Agile
Adopting Agile
 
ADDO19 - Automate or not from the beginning that is the question
ADDO19 - Automate or not from the beginning that is the questionADDO19 - Automate or not from the beginning that is the question
ADDO19 - Automate or not from the beginning that is the question
 
Intro to Lean UX with UserTesting
Intro to Lean UX with UserTestingIntro to Lean UX with UserTesting
Intro to Lean UX with UserTesting
 
DevOps in action - Azure DevOps
DevOps in action - Azure DevOpsDevOps in action - Azure DevOps
DevOps in action - Azure DevOps
 
When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"
When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"
When Management Asks You: “Do You Accept Agile as Your Lord and Savior?"
 
Ericriesleanstartuppresentationforweb2
Ericriesleanstartuppresentationforweb2Ericriesleanstartuppresentationforweb2
Ericriesleanstartuppresentationforweb2
 
Offshore IT Projects - Best Practices
Offshore IT Projects - Best PracticesOffshore IT Projects - Best Practices
Offshore IT Projects - Best Practices
 
Designing for efficiency.pdf
Designing for efficiency.pdfDesigning for efficiency.pdf
Designing for efficiency.pdf
 

More from Randy Shoup

Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityRandy Shoup
 
Breaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building TeamsBreaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building TeamsRandy Shoup
 
Managing Data in Microservices
Managing Data in MicroservicesManaging Data in Microservices
Managing Data in MicroservicesRandy Shoup
 
Effective Microservices In a Data-centric World
Effective Microservices In a Data-centric WorldEffective Microservices In a Data-centric World
Effective Microservices In a Data-centric WorldRandy Shoup
 
From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015Randy Shoup
 
Concurrency at Scale: Evolution to Micro-Services
Concurrency at Scale:  Evolution to Micro-ServicesConcurrency at Scale:  Evolution to Micro-Services
Concurrency at Scale: Evolution to Micro-ServicesRandy Shoup
 
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYEQCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYERandy Shoup
 
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...Randy Shoup
 

More from Randy Shoup (8)

Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
Breaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building TeamsBreaking Codes, Designing Jets, and Building Teams
Breaking Codes, Designing Jets, and Building Teams
 
Managing Data in Microservices
Managing Data in MicroservicesManaging Data in Microservices
Managing Data in Microservices
 
Effective Microservices In a Data-centric World
Effective Microservices In a Data-centric WorldEffective Microservices In a Data-centric World
Effective Microservices In a Data-centric World
 
From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015From the Monolith to Microservices - CraftConf 2015
From the Monolith to Microservices - CraftConf 2015
 
Concurrency at Scale: Evolution to Micro-Services
Concurrency at Scale:  Evolution to Micro-ServicesConcurrency at Scale:  Evolution to Micro-Services
Concurrency at Scale: Evolution to Micro-Services
 
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYEQCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
QCon New York 2014 - Scalable, Reliable Analytics Infrastructure at KIXEYE
 
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
QCon Tokyo 2014 - Virtuous Cycles of Velocity: What I Learned About Going Fas...
 

Recently uploaded

Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Cizo Technology Services
 
What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....kzayra69
 
Cyber security and its impact on E commerce
Cyber security and its impact on E commerceCyber security and its impact on E commerce
Cyber security and its impact on E commercemanigoyal112
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesPhilip Schwarz
 
英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作qr0udbr0
 
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfGOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfAlina Yurenko
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationBradBedford3
 
Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)Hr365.us smith
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)jennyeacort
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Angel Borroy López
 
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Natan Silnitsky
 
Best Web Development Agency- Idiosys USA.pdf
Best Web Development Agency- Idiosys USA.pdfBest Web Development Agency- Idiosys USA.pdf
Best Web Development Agency- Idiosys USA.pdfIdiosysTechnologies1
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtimeandrehoraa
 
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样umasea
 
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...stazi3110
 
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...OnePlan Solutions
 

Recently uploaded (20)

Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort ServiceHot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
 
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...
 
What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....What are the key points to focus on before starting to learn ETL Development....
What are the key points to focus on before starting to learn ETL Development....
 
Cyber security and its impact on E commerce
Cyber security and its impact on E commerceCyber security and its impact on E commerce
Cyber security and its impact on E commerce
 
Folding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a seriesFolding Cheat Sheet #4 - fourth in a series
Folding Cheat Sheet #4 - fourth in a series
 
英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作英国UN学位证,北安普顿大学毕业证书1:1制作
英国UN学位证,北安普顿大学毕业证书1:1制作
 
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdfGOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
 
How to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion ApplicationHow to submit a standout Adobe Champion Application
How to submit a standout Adobe Champion Application
 
Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)Recruitment Management Software Benefits (Infographic)
Recruitment Management Software Benefits (Infographic)
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanySuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte Germany
 
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
Call Us🔝>༒+91-9711147426⇛Call In girls karol bagh (Delhi)
 
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
Alfresco TTL#157 - Troubleshooting Made Easy: Deciphering Alfresco mTLS Confi...
 
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
 
2.pdf Ejercicios de programación competitiva
2.pdf Ejercicios de programación competitiva2.pdf Ejercicios de programación competitiva
2.pdf Ejercicios de programación competitiva
 
Best Web Development Agency- Idiosys USA.pdf
Best Web Development Agency- Idiosys USA.pdfBest Web Development Agency- Idiosys USA.pdf
Best Web Development Agency- Idiosys USA.pdf
 
SpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at RuntimeSpotFlow: Tracking Method Calls and States at Runtime
SpotFlow: Tracking Method Calls and States at Runtime
 
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
 
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
 
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
Tech Tuesday - Mastering Time Management Unlock the Power of OnePlan's Timesh...
 

Ten Lessons of the DevOps Transition

  • 1. Ten (Hard-Won) Lessons of the DevOps Transition Randy Shoup @randyshoup linkedin.com/in/randyshoup
  • 2. 1. Reorganize Teams Around Ownership • End-to-end Ownership o Small, cross-functional team owns application / service from design to deployment to retirement o Team has inside it all skill sets needed to do the job o Depends on other teams for supporting services o Able to move very rapidly and independently • “You build it, you run it” o The same team that builds the software operates the software o No separate maintenance or sustaining engineering team
  • 3. 1. Reorganize Teams Around Ownership • E.g., KIXEYE and MySQL o Development team wrote the SQL, issued all the queries o DBA / Ops team responsible for performance and uptime o Splitting ownership between teams was counterproductive and disruptive • Alternative strategies o Centrally-maintained persistence service OR o Customer manages its own persistence
  • 4. 2. Lose the Ticket Culture Ticket Culture Ownership Culture Do what is asked for Do what is needed One-way communication Two-way collaboration Goal is to close the ticket Goal is product success Reactive approach Proactive approach Reinforces silos Reinforces collaboration Prioritizes process Prioritizes results
  • 5. 3. Replace Approvals With Code • Reduce or eliminate approval bodies o E.g., eBay Architecture Review Board o (-) Too late o (-) Too slow o (-) Too disengaged from details • Package expertise in code o Smart, experienced people build their knowledge into code o Teams with specialized skills (databases, security, compliance, etc.) provide a service, library, or tool
  • 6. 3. Replace Approvals With Code • E.g., Security at Google o Provide secure foundations by maintaining lower-level libraries and services o Provide self-service penetration tests, vulnerability assessments, etc.
  • 7. The easiest way to “enforce” a standard practice is with working code.
  • 8. 4. Enforce a Service Mentality • Vendor-Customer Discipline o Service team is a vendor; the products are its customers o Service is useful only to the extent it provides value to its customers • Customer can choose to use service or not (!) o Customer team is responsible for deciding what is best for their use case o Use the right tool for the right job • Provides powerful incentives o Service must be *strictly better* than the alternatives of build, buy, borrow
  • 9. 5. Charge for Usage • Charge customers for *usage* of the service o Aligns economic incentives of customer and provider o Motivates both sides to optimize efficiency • Free usage leads to waste o No incentive to control usage or find more efficient alternatives • E.g., App Engine usage at Google o Charging particularly egregious internal customer led to 10x reduction in usage
  • 10. 6. Prioritize Quality • Quality, Performance, and Reliability are “Priority-0 features” o “Stop the line” if there is a degradation o Equally important to users as product features or engaging user experience • Developers write tests and code together o Continuous testing of features, performance, load o Confidence to make risky changes • “Slow down to speed up” o Catch bugs earlier, fail faster
  • 11. 6. Prioritize Quality • E.g., Development Process at Google o Code reviews before submission o Automated tests for everything o Single searchable source code repository  Internal Open Source Model o Not “here is a bug report” o Instead “here is the bug; here is the code fix; here is the test that verifies the fix” 
  • 12. 7. Start Investing in Testing • Write functional tests around a component o If you can only write a few tests, they should be meaningful ones o End-to-end tests exercise more meaningful customer-visible capabilities than unit tests • Fail any build that breaks a test • Keep ratcheting up the tests o For every new feature, add tests for that feature o For every new bug, add a test that reproduces the bug and verifies the fix
  • 13. 8. Actively Manage Technical Debt • Maintain sustainable and well-understood level of debt o Denominated in engineering effort to fix o Plan for how and when you will pay it off o Track feature work vs. accrued debt over time • “Don’t have time to do it right” ? o WRONG  – Don’t have time to do it twice (!) o The more constrained you are on time and resources, the more important it is to do it solidly the first time
  • 14. Vicious Cycle of Technical Debt Technical Debt “No time to do it right” Quick- and-dirty
  • 16. 9. Share On-call Duties • All members of the team rotate on-call responsibilities o Strongest motivator to build in solid monitoring and diagnosis capabilities o Best way to learn the real-world behavior of the system o Best way to develop empathy for customers and other team members • Train via on-call “apprenticeship” o 1. Apprentice starts as secondary on-call, experienced engineer is primary o 2. Apprentice is primary, experienced engineer is secondary o 3. Apprentice graduates
  • 17. 10. Make Post-Mortems Truly Blameless • Overcoming blame culture takes work o Institutional memory of blame is long o E.g., Initial post-mortems at KIXEYE elicited tons of fear • Constantly reinforce learning over blame o When you say “blameless”, you have to really mean it (!) o Don’t ask “what did you do?”, ask “what did you learn?”
  • 18. 10. Make Post-Mortems Truly Blameless • Open and Honest Discussion o Document exactly what happened o What went right o What went wrong • Focus on Learning and Improvement o How should we change process, technology, documentation, etc. o How could we have automated the problems away? o How could we have diagnosed more quickly? • Take fear and personalization out of it  Engineers will compete to take personal responsibility (!)  “Finally we can fix that broken system” 
  • 19. Top Five Takeaways • 1. Reorganize Teams Around Ownership • 2. Replace Approvals With Code • 3. Prioritize Quality • 4. Actively Manage Technical Debt • 5. Make Post-Mortems Truly Blameless
  • 20. What I Could Use Help With • Encouraging leaders to lose the blame culture • Measuring productivity in a principled way • Overcoming resistance to taking the pager
  • 21. Thank You! • @randyshoup • linkedin.com/in/randyshoup