5. The Problem In A Nutshell
Everything needs software.
Software runs on a server to become a
service.
Delivering a service from inception to its
users is too slow and error-prone.
There are internal friction points that
make this the case.
This loses you money. (Delay = loss)
Therefore IT is frequently the bottleneck
in the transition of “concept to cash.”
6. Symptoms
• Defects released into production, causing outage
• Inability to diagnose production issues quickly
• Problems appear in some environments only
• Blame shifting/finger pointing
• Long delays while dev, QA, or another team waits
on resource or response from other teams
• “Manual error” is a commonly cited root cause
• Releases slip/fail
• Quality of life issues in IT
7. Why Does This Problem Exist?
“Business-IT Alignment?”
The business has demanded the
wrong things out of IT
Cost sensitive
Risk averse
IT has metastasized over time into a
form to give the business what it’s
said it wants
Centralized and monolithic
Slow and penny wise, pound foolish
9. DevOps Defined
DevOps is the practice of operations
and development engineers
participating together in the entire
service lifecycle, from design
through the development
process to production support.
DevOps is also characterized by
operations staff making use many of
the same techniques as developers
for their systems work.
10.
11. DevOps Defined
DevOps is the practice of operations
and development engineers
participating together in the entire
service lifecycle, from design
through the development
process to production support.
DevOps is also characterized by
operations staff making use many of
the same techniques as developers
for their systems work.
12.
13. DevOps History In 60 Seconds
ITIL, ITSM, ESM, etc. underdeliver in IT from 1989 on
Agile comes to the developer world in 2001
Lean comes to the developer world in 2003 (more slowly)
O’Reilly Radar “Operations: The New Secret Sauce” in 2006
Agile Infrastructure discussions start in Europe circa 2007
Patrick Debois and Andrew Schafer meet up at Agile 2008
O’Reilly Velocity Conference starts 2008
Velocity 2009, seminal John Allspaw “10+ Deploys Per Day:
Dev and Ops Cooperation” presentation
Patrick Debois and Kris Buytaert put together first
DevOpsDays in Ghent in 2009. Many more follow
Lean influences enter DevOps via startup culture
Large companies start branding DevOps “solutions”
16. DevOps Principles
• The Three Ways
• Systems Thinking
• Amplify Feedback Loops
• Culture of Continual Experimentation
• CAMS
• Culture – People > Process > Tools
• Automation – Infrastructure as Code
• Measurement – Measure Everything
• Sharing – Collaboration/Feedback
• Informed by the values in the Agile
Manifesto and Lean Theory of Constraints
17. DevOps Practices
• Version Control For All
• Automated Testing
• Proactive Monitoring and Metrics
• Kanban/Scrum
• Visible Ops/Change Management
• Configuration Management
• Incident Command System
• Continuous Integration/Deployment/Delivery
• “Put Developers On Call”
• Virtualization/Cloud/Containers
• Toolchain Approach
• Transparent Uptime/Incident Retrospectives
20. Add Ops Into Dev
Enhance Service Design With Operational
Knowledge
Reliability
Performance
Security
Test Them
Build Feedback Paths Back from Production
Monitoring and metrics
Postmortems
Foster a Culture of Responsibility
Whether your code passes test, gets deployed, and stays
up for users is your responsibility – not someone else’s
Make Development Better With Ops
Productionlike environments
Power tooling
21.
22. Accelerate Flow To Production
Reduce batch size
Automated environments mean identical
dev/test/prod environments
Create safety through automation
Continuous Integration/Testing
Automated Regression Testing
Continuous Delivery
Continuous Deployment
Feature Flags (A/B testing)
Security Testing
23.
24. Add Dev Into Ops
• Don’t do tasks for people. Build tools so they
can do their own work.
• Monitoring/logging/metrics feeds back into dev
(and the business)
• Blameless Incident Postmortems
• Developers Do Production Support/Empower
Ops Acceptance
25. Grass Roots Checklist
Find ways to collaborate – involve others early
Find ways to automate and make self-service
Become metrics driven
Learn new things, continually improve
Understand the larger business goals, metrics,
and priorities you support
Communicate
Work in parallel with small batches
Allow refactoring
Prove the business value to management
26. Management Checklist
• Experiment – choose a test case as a pilot
• Then document and spread best practices
• Empower your teams, but guide their values
• Metrics are your friend – demand measurable
outcomes
• Don’t accept excuses when the old baseline isn’t
good enough
• Fail fast, continually improve
• Build on small successes to gain broad support for
more substantive change.
• Align roles and responsibilities across groups –
enable collaboration even if it seems “inefficient”
27. Things Not To Do
Only Token Gestures
“Ops team, change your name to DevOps team!”
“Put DevOps in those job titles!”
Only Implement Tools
Changing tools without changing tactics leaves the
battlefield strewn with bodies
Create More Silos
Devalue Operations Or Development Knowledge
Anything You’re Not Measuring The Impact Of
28. Does It Really Help?
•2014 State of DevOps Report (9200
surveyed) measured correlation
between high performing
organizations and DevOps practice
adoption
• Lead time to changes down
• MTTR up
• No alteration in change fail rate
29. Core DevOps Research List
• Gene Kim’s Visible Ops
• Tom Limoncelli’s The Practice Of Cloud System Administration
• Gene Kim’s The Phoenix Project (modeled on Goldratt’s The Goal)
• Jez Humble’s Continuous Delivery
• Michael Nygard’s Release It!
• Gene Kim’s The DevOps Cookbook (coming soon-ish)
• Various Mary and Tom Poppendieck Lean Software Development
Books
• Velocity Conference (velocityconf.com)
• DevOpsDays Unconferences – There’s one near you!
(devopsdays.org)
• DevOps Weekly newsletter (devopsweekly.com)
• DevOps Café Podcast (devopscafe.com)
• The Twelve Factor App (12factor.net)
• The Agile Admin (theagileadmin.com)
Editor's Notes
Who are you? Manager? Developer? Sysadmin? Something else? Are you interested in the answer to these questions?
Hi, I’m Ernest. I am not a mall cop. I’ve been in IT since 1993. I’ve been a developer, a sysadmin, an architect, an IT manager, a development manager, a DevOps manager, and a product manager. I’ve worked in huge IT organizations and small SaaS startups. I’ve led three DevOps transformations so far, and have seen compelling results each time.
I asked Google what DevOps was. Apparently it’s pretty complicated. (And utterly lacking in graphic design capabilities.) I’m not going to tell you anything someone else hasn’t said in this presentation, but what I am going to do is synthesize it from a hundred blogs and conference proceedings and tweets and IRC discussions into something normal folks can understand.
I know this sounds pretty basic, but bear with me so we’re all on the same page.
Nowadays, everything you want to do is either software, or requires software to do. You want to do sales bundling a different way, you want to gather usage data on your trucks, you want to optimize your manufacturing line. It all requires software, running on servers – in ITSM-speak we call these “services.” Email is a service, your e-commerce Web site is a service, Twitter is a service.
Here’s some hints that you have this root-cause problem.
Remember the year 2000? When CIO magazine was still in paper form, many inkwells were spilled about this topic.
But of course, competitive pressure made this untenable. So the business started pushing on the intake side of IT – development. Dev groups frequently reorganized to match LOBs and focused on innovation, and did things like uptake agile software development. But this just created the same problem in miniature inside IT, with the responsibility for reliability and cost siloed into Operations.
A lot of the extant DevOps writing focuses on this impedance mismatch, with Dev wanting to push shaky code ever faster and Ops wanting more guarantees of stability and more process checks, since they are responsible for the stability and cost.
This results in both a flow problem and a quality problem, which anyone with a background in manufacturing will understand immediately, it’s a basic “The Goal” theory of constraints problem.
DevOps definition – part one.
When I was a systems engineering manager at NI in Austin, we had great staff and did as much as we could, but were still “the bottleneck.” Every month we ran these Web software releases where 30-50 people would be on line from 7 PM Friday well through Saturday morning. We had hundreds of on-call pages a week some weeks. Finally one day I came to the realization that there was something fundamentally wrong with our approach. The overall way the software pipeline was constructed was designed in a way to never flow smoothly. Our agile, line-of-business-aligned development teams were crashing against a big, monolithic, infrastructure organization horizontally striped by technology specialty. And rather than understanding the theory of constraints, our organization – like most organizations – was just pushing harder into the hopper and hoping more sausage would come out the end. But the sausage that emerged… Was not fit for consumption.
I went to our business owner and explained the problem to him. We then got Dev and Ops together and we put together a five-point set of goals that he would hold both dev and ops responsible for.
DevOps definition – part two.
The problem’s not just “outside” ops. To be honest, I found that in ops we had let a lot of computing best practices pass us by. Use of source control, testing, and the like are just plain better, and most system administrators didn’t do any of that.
I came to this realization when we got a new VMWare farm in at NI. It took my team 6 weeks minimum to get in a new server – for we had to navigate all the other horizontal IT Infrastructure teams as well. Data center, networking, UNIX or Windows, everyone had their own processes and checklists. The sales people had gleefully demonstrated how “you can have a new server in 15 minutes!” This got my time down to getting a new server to 4 weeks. All the same delays, just with the Dell procurement turnaround removed from the process.
“We’re turning 15 minutes into 4 weeks.” That doesn’t pass the smell test. Blah blah standardization blah blah cost blah blah specialization – WHERE THE HELL IS MY SERVER?
And now you’re caught up. I had been going to Velocity and watching the Agile Infrastructure list, and then got involved when Damon Edwards and John Willis put on OpsCamp Austin in 2010 and DevOps was the hot topic.
Culture. You may have heard this already. Sounds hippy-dippy. What is “culture?”
Culture (/ˈkʌltʃər/) is, in the words of E.B. Tylor, "that complex whole which includes knowledge, belief, art, morals, law, custom, and any other capabilities and habits acquired by man as a member of society." (Tylor 1871:1)
One of the things that makes learning about DevOps difficult is that different discussions slot in at different parts in the conceptual hierarchy around DevOps. I’m not going to go through these in depth, they are just examples – but when you hear someone talking about something that “is DevOps” or “is not DevOps” it can bring clarity to understand what conceptual level they are talking about.
You may also see this stated as People/Process/Tools.
At the top level you have the cultural and conceptual change required to set the vision and values for DevOps.
The key is a high trust, shared risk culture with a win-win relationship between dev and ops (and everyone else).
The Three Ways are Gene Kim’s way of communicating DevOps principles; John Willis coined CAMS.
Then you get into specific DevOps practices. No specific ones of them are necessary or sufficient per se, it just depends which you will realize value from. Our first DevOps implementation omitted all CI/CD because that was not a problem we had. This is by no means a complete list.
These need to flow from the principles – I’ve been on the receiving end of very non-DevOps CM implementations, for example.
DevOps Tools, also known as “Chef or Puppet?” I am not covering tools in this talk, except to say:
There are a lot of great tools that can help you implement some of the DevOps practices
Putting tools first as the solution to a pervasive problem will work about as well as it ever has.
ITSM, back from the dead to our rescue. ITIL had a good conceptual model, but most proposed implementations approached the problem from the wrong mindset. A service can be described has having three phases in one big feedback loop – Design, Transition, and Operation. When we talk in DevOps we often shorthand these as “Dev,” “Release,” and “Ops” even though that’s usually not strictly correct. The phases don’t map one-to-one with skill sets, and there are other disciplines like QA, security, network engineers, etc. that participate in these. But Service DesTranOps isn’t as catchy.
These are sometimes ironically called nonfunctional requirements. Because if you neglect them, your service is nonfunctional.
There are many ways to do this. You can embed operations experts into development teams, agile style. I have done that several times. You can make developers support their own app until it’s stable enough, Google SRE style. The most important thing is to work with people, not against them.
When managing a systems team, the developers rolled out a new version that needed 10x the amount of system resources it was using previously. “Can’t you just add more hardware,” they asked? “Not ten times as much hardware.” They then fixed the issue in the next release. Decisions made in the design phase impact performance, reliability, and security by orders of magnitude and is obviously the highest leverage point to make improvements.
This can be hard, when we had joint dev and ops spec review meetings the devs were initially put off by the “weird ops questions.” But if you power through that till everyone’s speaking the same language, you can start to solve those problems.
Automate your change control process.
Even if you can’t get to continuous deployment, getting the increment down from typical quarterly, six week, or monthly releases reduces the number of things to go wrong. I was brought in to one SaaS company as a release manager to get their release cadence from an ever-slipping six weeks down to two weeks. This required test automation, deployment automation, and process discipline. We got on a regular cadence and it went so well that we then moved to weekly releases with no fanfare later.
There was a smaller payload to test, so it was tested more thoroughly. If something went wrong, there was less payload to troubleshoot. If there was a bug, there was another release right around the corner. Dates didn’t slip, items just made a release or didn’t. Order came from chaos.
In other words, enhance your service operation phase with an understanding of development techniques.
Are you implementing this “bottom up?” Don’t go tool crazy, think about what you’re doing, and be able to track and prove the improvement to get more support.
Or are you looking to implement “top down?” Here’s some steps to take.
Remember, you are looking for the highest overall flow of successfully delivered services for your limited resources. One specific thing that works here is reorganizing so necessary operations expertise is embedded in or aligned with the business/dev flow – basic queueing theory (hint: M/M/1 Markov queues) will show that having multiple consuming groups engaged with one centralized group to get what they need is suboptimal and as the load increases on that group, the wait time approaches infinity.
DevOps can fail for the reason most other things fail – because you take the easy way out and give it lip service instead of doing the hard work.
Cultural transformation is hard, and shared responsibility takes commitment
I personally like The Principles of Product Development Flow but it’s pretty hardcore.
Many more specialized conferences have flourished in the last couple years – SREConf, Monitorama, SCALE, SURGE, re:invent…
Questions?
Come to DevOpsDays Austin! And look up San Antonio DevOps on Meetup.com!