3. The Problem
• Agile Continuous Delivery perceived as incompatible with
Operations team ITIL type change management
methodology:
• Need for specific versions of the application and
supporting tools
• Change management process requires detailed specifics of
what's in a "release" in order to assess possible impact
• Multiple environments that need to be maintained for
integration, staging, performance, etc.
• Requirement to perform granular upgrades to existing
environments
4. Negotiation Deadlock
• Always shipping from trunk - Lack of release
branches
• New functionality exposed by feature flags -
Lack of discrete features or fixes per release
• Package version availability (or rather lack of)
i.e who cares about version X in 6 months?
• Reliance on Rolling Forward vs Back
5. Valid questions and
possible assumptions
• Will version X be able inY Months (With
multiple releases per day)?
• Should the managing software versions and
a definitive software library across multiple
environments be a full time job?
6. Conflicting priorities
and incentives
• Customers want features
• Business wants to quickly deliver features to
customers in a reliable and stable manner
• Developers want change to enable features
• Operations want stability and minimal
changes
7. But...
• We build a release candidate on every
successful commit
• New functionality gets added all the time
• A lot can change if you don’t ship regularly
• If you skip xxx revisions deployment risk
increases
8. The cost of long durations
between releases
• Big releases are risky!
• Big releases reduce code value (code depreciation)
(http://martinfowler.com/bliki/FrequencyReducesDifficulty.html)
9. Big Changes are scary
• If Big Changes are scary lets split them into
small batches
• Small batches mean faster feedback
• Small batches mean problems are
instantly localized
• Small batches reduce risk
• Small batches reduce overhead
(http://www.startuplessonslearned.com/2009/02/work-in-small-batches.html)
10. Economic benefits of small
batch sizes
• Donald G. Reinertsen’s “The Principles of Product Development Flow”, page 121
(http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-improve-flow/)
11. Doing it more often requires a continuous
integration or delivery pipeline
Doing it more often
12. • Example IT Change Management Process
ITIL Change Management
Process
13. Gap Analysis
• How do we get the artifacts supplied by
our continuous deployment pipeline
integrated into the change management
process?
15. Additional Tooling
• As part of our Continuous Integration
process we deliver RPM Packages
• RPM Packages are hosted in a package
repository and for drop off to our demo
environments and pulp.
16. Why RPM’s?
• Our selected OS's default package manager
• Deployment is easy for Traditional Operations
Teams
• We are packaging a package but the incremental
work performed to create them more than paid
for itself in terms of communications overhead for
release coordination
• We want to package almost everything in a RPM
(excluding configuration) e.g. Documentation,
Software,Admin support scripts, etc...
17. Pulp?
• Pulp is a platform for managing repositories of content, such
as software packages, and pushing that content out to large
numbers of consumers
• Pulp can walk software packages through development,
testing, and stable repositories, pushing those updates out to
client machines as they get promoted.
18. Definitive Media
Library
• Pulp becomes our ITILv3 Definitive Media
Library
• Vendor our upstream packages and
dependencies
• It also has support for mirroring puppet
forge modules
24. Conclusion
• Releases can flow through the system in a
manner that fits all parties
• As confidence and trust grows we can move
from normal to standard pre-approved releases
further increasing deployment velocity
• Working towards making production releases
boring events rather than fire drills
• It adds predictability since the process flow is
shown from code creation to production
deployment
• Shared responsibility between all teams
involved in getting releases into production