Given at #DeliveryConf Seattle

1. What is quality, and how
do we build it in?
@maryamumar | @jezhumble | #DeliveryConf

2. Agenda
● Introduce Thought Machine
● How to get started with improving quality
● Selecting metrics
● Evolving metrics based on what we learn
@jezhumble
@maryamumar

3. TLDR
● Quality is subjective - what to focus on depends on many factors
● Start by creating a vision, selecting some metrics, and measuring
● What you measure will evolve, and will determine what teams work on
@jezhumble
@maryamumar

4. Please fix
Quality!
Clients are unhappy
Issues are found late
Releases are delayed
Team feels stressed

6. Common mistakes we have observed
● Stop work and build a comprehensive automated test suite
● Add new process and approvals to try and prevent escaped defects
● Create a new “quality” function which sits downstream of development
● Underinvestment in infrastructure and tools
@jezhumble
@maryamumar

7. Create a Plan
But where do you start?*
*Leading Quality - Ronald Cummings-John, Owais Peer

8. Vision Statement
As an <engineering team/customer>
When <there are any changes to production code>
Then <every merge to the codebase is releasable>
And <there are no known regressions in the system>
@maryamumar

9. Assuring Quality enables teams to drive
for customer satisfaction at a sustainable
pace.

10. Who are your Stakeholders?
● CEO/CTO
● Tech leads
● Testing team
● Incident Management
● Release Management
Questions to ask:
● How do you know quality was bad?
● Escaped defects?
● What’s your code coverage like?
● Do you have the right processes in place?
@maryamumar

11. Software Delivery Performance
@jezhumble https://cloud.google.com/devops/state-of-devops/

12. How to get better
https://bit.ly/dora-bfd@jezhumble

13. Areas to Improve
Releases Product Quality Process Quality
@maryamumar

14. Releases
- Create visibility landscape for releases
- Monitor issues found post-release vs pre-release
- How good is your regression cycle?
- Release notes
- Changes
- Known issues; when to expect fixes
- How easy is it to create deployables?
- How quickly can you deploy?
@jezhumble
@maryamumar

16. Product Quality
@jezhumble
@maryamumar
MTTG Build FailuresBuild Time Deploy Time
Average Time
in Status
PR Review PR commit rate
per Sprint
%age of Flaky
Tests
Ticket
Fluctuation
from QA to Dev
Test Coverage

17. Product Quality (contd.)
@jezhumble
@maryamumar
Defects found
in Sprint vs
Escaped
Defects
Team FeedbackDefects found
via Automation
via Exploratory
testing per
Feature
Bug resolution
time per
Severity level

20. Time from check-in to deployed in test environment

21. SATs vs Defects per Team

22. High trust culture
Westrum, “A Typology of Organizational Cultures” | http://bmj.co/1BRGh5q
how organizations process information

23. Psychological safety
“Wherever there is fear, you will get wrong figures”
— W Edwards Deming
@jezhumble
@maryamumar

24. https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/
Culture of psychological safety
● Predicts software delivery performance
and organizational performance
● Implement by adopting continuous
delivery and lean product management
practices
@jezhumble

25. 2018 State of DevOps Report: cloud.google.com/devops
SDO Performance
Trust
Software
Delivery
Performance
Availability
Organizational
Performance
Voice
Westrum organizational culture
Autonomy
Retrospectives
Climate for Learning
@jezhumble

26. How to make metrics safe
● Teams involved in creating vision
● Consult with teams when deciding what to measure
● Testing team makes quality visible
● Let teams set their own targets
● Regular retrospectives on what is being measured
@jezhumble

27. Conclusion
● Quality is subjective - what to focus on depends on many factors
● Start by creating a vision, selecting some metrics, and measuring
● What you measure will evolve, and will determine what teams work on
© 2020 Google LLC & Thought Machine Group Ltd

28. Parking lot

29. Continuous testing
● Continuously reviewing and improving test suites to better find defects and keep complexity and cost
under control
● Allowing testers to work alongside developers throughout the software development and delivery
process
● Performing manual test activities such as exploratory testing, usability testing, and acceptance testing
throughout the delivery process
● Having developers practice test-driven development by writing unit tests before writing production
code for all changes to the codebase
● Being able to get feedback from automated tests in less than ten minutes both on local workstations
and from a CI server
2018 State of DevOps Report | cloud.google.com/devops

30. Continuous delivery
The ability to get changes—features,
configuration changes, bug fixes,
experiments—into production or into the hands of
users safely and quickly in a sustainable way.
2018 State of DevOps Report | cloud.google.com/devops

31. Technical debt quadrant
https://martinfowler.com/bliki/TechnicalDebtQuadrant.html