Presentation by Damon Edwards, co-founder of Rundeck, at DevOps Enterprise Summit in San Francisco, November 13, 2017 See a Demo of Rundeck Enterprise : https://www.rundeck.com/see-demo --or-- Download Rundeck Open Source here: https://rundeck.com/open-source Connect: Stack Overflow community: https://stackoverflow.com/questions/tagged/rundeck Github: https://github.com/rundeck/rundeck/issues Twitter: https://twitter.com/Rundeck Facebook: https://www.facebook.com/RundeckInc/ LinkedIn: www.linkedin.com › company › rundeck-inc

1. Keeping your DevOps Transformation
from Crushing Your Ops Capacity
Damon Edwards
@damonedwards

2. Damon
Edwards
Ops Improvement
DevOps Consulting
Ops Tools
Community

3. This talk is about what happens after deployment
Deploy
Everything
Else
What does Ops do with it’s time?

4. Let’s start with an (unfortunately) typical incident…
Download the PDF here: https://rundeck.co/incident_does2017

5. And people still wonder where the time goes.
DevOps, Containers,
Continuous Deployment.. Why
aren’t we getting more done?
Executive Team

6. It’s only getting worse
The Operations Squeeze
Introduction
“The Operations Squeeze”
Go faster! Be flexible! Lock it down!
Improved Quality
Shorter Time-to-Market
Fast Feedback
From Users
Availability Auditing
Security Compliance
Dev Ops
Ops
“The Ops Squeeze”

7. It’s only getting worse
The Operations Squeeze
Introduction
“The Operations Squeeze”
Go faster! Be flexible! Lock it down!
Improved Quality
Shorter Time-to-Market
Fast Feedback
From Users
Availability Auditing
Security Compliance
Dev Ops
Ops
More errors
More delays
Less capacity
Less flexibility
“The Ops Squeeze”

8. Use a “Lean” lens to analyze operations activity

9. Lean: Empower those closest to the issue
escalate escalate
1° 2° 3°
escalate
4°

10. Lean: Empower those closest to the issue
escalate escalate
1° 2° 3°
escalate
4°
Push the ability to take action this direction

11. Lean: Empower those closest to the issue
escalate escalate
1° 2° 3°
escalate
4°
Push the ability to take action this direction
But what gets
in the way?

12. Lean: Empower those closest to the issue
escalate escalate
1° 2° 3°
escalate
4°
Push the ability to take action this direction
SilosBut what gets
in the way?

13. Backlog Context
I need X
Backlog
I do X
Requests
for X
Silo A
Priorities
Context
Priorities
Silo B
Tools Tools
Silos ruin everything

14. Backlog Context
I need X
Backlog
I do X
Requests
for X
Silo A
Priorities
Context
Priorities
Silo B
Tools Tools
Silos ruin everything

15. How to spot silos? Look for Ticket-Driven Request Queues
Team A
(Dev)
Team B
(Ops)
Ticket
System
??

16. Team A
(Dev)
Team B
(Ops)
Ticket
System
??
Silo Builder
How to spot silos? Look for Ticket-Driven Request Queues

17. Team A
(Dev)
Team B
(Ops)
Ticket
System
??
Silo Builder Snowflake Maker
How to spot silos? Look for Ticket-Driven Request Queues

18. Request queues proven to have a huge economic impact

19. All of those little delays end up being really expensive
RevenueperWeek
Time
COST OF DELAY Actual Revenue
Market: Long Lifetime Value, Peak Unaffected by Delay
Opportunity Ready

20. So lets get rid of silos & request queues!

21. Popular: Replace Silos with Cross Functional Team
Dev/Test Release OperatePlanning

22. Popular: Replace Silos with Cross Functional Team
Dev/Test Release OperatePlanning
Cross-Functional Teams
Cross-Functional Teams
Cross-Functional Teams

23. Dev/Test Release OperatePlanning
Popular: Replace Silos with Cross Functional Team
Cross-Functional Teams
Cross-Functional Teams
Cross-Functional Teams

24. EnvironmentsDBAs Network Security NOC
Dev/Test Release OperatePlanning
Popular: Replace Silos with Cross Functional Team
Cross-Functional Teams
Cross-Functional Teams
Cross-Functional Teams

25. EnvironmentsDBAs Network Security NOC
Dev/Test Release OperatePlanning
Team A
(Dev)
Team B
(Ops)
Ticket
System
??
Popular: Replace Silos with Cross Functional Team
Cross-Functional Teams
Cross-Functional Teams
Cross-Functional Teams

26. Team A
(Dev)
Team B
(Ops)
Ticket
System
??
Silo Builder Snowflake Maker
Get rid of the remaining ticket-driven request queues…

27. … by replacing with Operations as a Service design pattern
Team A
(Dev)
Team B
(Ops)
Ticket
System
Operations
as a
Service
Actual Exceptions
On
Demand
On
Demand

28. … by replacing with Operations as a Service design pattern
Team A
(Dev)
Team B
(Ops)
Ticket
System
Operations
as a
Service
Actual Exceptions
On
Demand
On
Demand

29. Change how your organization thinks
about automated procedures…

30. Automated procedures are comprised of three parts
Definition of the automated procedure
Execution of the automated procedure
Governance of the automated procedure
Define
Execute
Govern

31. Automated procedures are comprised of three parts
Definition of the automated procedure
Execution of the automated procedure
Governance of the automated procedure
Define
Execute
Govern
(security, oversight, compliance, etc.)

32. Traditional Ops Silo
Define
Execute
Govern
“Consumers of Ops”
(Dev, QA, Release, NOC, Security, etc.)
Ops

33. Rigid Self-Service
Define
Execute
Govern
“Consumers of Ops”
(Dev, QA, Release, NOC, Security, etc.)
Ops

34. Define
Execute
Govern
Execute
“Consumers of Ops”
(Dev, QA, Release, NOC, Security, etc.)
Ops
Rigid Self-Service (limited)

35. High-Velocity Handoffs
Define
Govern
Execute
“Consumers of Ops”
(Dev, QA, Release, NOC, Security, etc.)
Ops

36. Self-Service Operations
Define
Govern
Execute
“Consumers of Ops”
(Dev, QA, Release, NOC, Security, etc.)
Ops

37. Self-Service Operations
Define
Govern
Execute
Govern
“Consumers of Ops”
(Dev, QA, Release, NOC, Security, etc.)
Ops

38. fdfd
Operations as a Service
Operations
as a
Service
ED G
Team B
(Ops)
Vet
Procedures
Define
Policies
Execute
On Demand
Team A
(Dev)
Define
Procedures
Execute
On Demand

39. fdfd
Operations as a Service
Move definition, execution, and governance to where you
get the most effective use of labor and best flow of work
Operations
as a
Service
ED G
Team B
(Ops)
Vet
Procedures
Define
Policies
Execute
On Demand
Team A
(Dev)
Define
Procedures
Execute
On Demand

40. fdfd
Lean: Standardize to eliminate waste and enable improvement
Operations
as a
Service
ED G
Team B
(Ops)
Vet
Procedures
Define
Policies
Execute
On Demand
Team A
(Dev)
Define
Procedures
Execute
On Demand
Collaborate to create standard
operating procedures

41. Example Operations as a Service Platform
#! ! "# $
Scripts APIs Tools Cloud VMs Containers
Orchestration
Collect and
Process Output
Infrastructure
details
Config.
Man.
CMDB
Monitor.
Metrics
Cloud
Corp
Directory
Authentication
ITSM Tickets
>_
● Collaborate on Standard Procedures
● Manage Access & Governance policies
● Execute Actions & Share Visibility
Web GUI API CLI
Operations as a Service Platform

42. Example Operations as a Service Platform
#! ! "# $
Scripts APIs Tools Cloud VMs Containers
Orchestration
Collect and
Process Output
Infrastructure
details
Config.
Man.
CMDB
Monitor.
Metrics
Cloud
Corp
Directory
Authentication
ITSM Tickets
>_
● Collaborate on Standard Procedures
● Manage Access & Governance policies
● Execute Actions & Share Visibility
Web GUI API CLI
Operations as a Service Platform

43. Example Operations as a Service Platform
#! ! "# $
Scripts APIs Tools Cloud VMs Containers
Orchestration
Collect and
Process Output
Infrastructure
details
Config.
Man.
CMDB
Monitor.
Metrics
Cloud
Corp
Directory
Authentication
ITSM Tickets
>_
● Collaborate on Standard Procedures
● Manage Access & Governance policies
● Execute Actions & Share Visibility
Web GUI API CLI
Operations as a Service Platform

44. Example Operations as a Service Platform
#! ! "# $
Scripts APIs Tools Cloud VMs Containers
Orchestration
Collect and
Process Output
Infrastructure
details
Config.
Man.
CMDB
Monitor.
Metrics
Cloud
Corp
Directory
Authentication
ITSM Tickets
>_
● Collaborate on Standard Procedures
● Manage Access & Governance policies
● Execute Actions & Share Visibility
Web GUI API CLI
Operations as a Service Platform

45. Example Operations as a Service Platform
#! ! "# $
Scripts APIs Tools Cloud VMs Containers
Orchestration
Collect and
Process Output
Infrastructure
details
Config.
Man.
CMDB
Monitor.
Metrics
Cloud
Corp
Directory
Authentication
ITSM Tickets
>_
● Collaborate on Standard Procedures
● Manage Access & Governance policies
● Execute Actions & Share Visibility
Web GUI API CLI
Operations as a Service Platform

46. Let’s look at some companies who are
leveraging this design pattern.

49. Mark
Maun
Jody
Mulkey
Justin
Dean

50. 90% Reduction in MTTR
50% Reduction in escalations
55% Reduction of overall support costs

52. 1. New org, support, and escalation model

53. escalate
1° 2° 3° 4°
escalate escalate
1. New org, support, and escalation model

54. escalate
1° 2° 3° 4°
escalate escalate
1. New org, support, and escalation model
EMT ER Trauma
Surgeon
Specialist
Surgeon
TOC (NOC) SRE
Production Eng.
Scrum Teams
Data Services
Platform Eng.
Global Network
> 15 min > 30 min > 60 min

55. 2. Key: Push the ability to take action closest to the problem
escalate
1° 2° 3° 4°
escalate escalate
1. New org, support, and escalation model
EMT ER Trauma
Surgeon
Specialist
Surgeon
TOC (NOC) SRE
Production Eng.
Scrum Teams
Data Services
Platform Eng.
Global Network
> 15 min > 30 min > 60 min

56. 2. Key: Push the ability to take action closest to the problem
escalate
1° 2° 3° 4°
escalate escalate
1. New org, support, and escalation model
EMT ER Trauma
Surgeon
Specialist
Surgeon
TOC (NOC) SRE
Production Eng.
Scrum Teams
Data Services
Platform Eng.
Global Network
> 15 min > 30 min > 60 min
3. Longterm investment in operability
(deployment, configuration, monitoring, automated runbooks)

57. “Support at the Edge”
Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ
http://rundeck.org/stories/mark_maun.html

58. “Support at the Edge”
Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ
http://rundeck.org/stories/mark_maun.html
• Automated Ops procedures written/
vetted by the delivery teams

59. “Support at the Edge”
Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ
http://rundeck.org/stories/mark_maun.html
• Automated Ops procedures written/
vetted by the delivery teams
• Ops remained in full control of what
can run and security policy

60. “Support at the Edge”
Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ
http://rundeck.org/stories/mark_maun.html
• Automated Ops procedures written/
vetted by the delivery teams
• Ops remained in full control of what
can run and security policy
• Empowered NOC and other support
teams with self-service ops tasks

61. “Support at the Edge”
Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ
http://rundeck.org/stories/mark_maun.html
• Automated Ops procedures written/
vetted by the delivery teams
• Ops remained in full control of what
can run and security policy
• Empowered NOC and other support
teams with self-service ops tasks
• Empowered developers with limited
self-service operations

62. “Support at the Edge”
Sources: https://www.youtube.com/watch?v=_hr4KiB19bQ
http://rundeck.org/stories/mark_maun.html
• Automated Ops procedures written/
vetted by the delivery teams
• Ops remained in full control of what
can run and security policy
• Empowered NOC and other support
teams with self-service ops tasks
• Empowered developers with limited
self-service operations
• Combined with new incident response
and escalation model

63. Fortune 100 Manufacturing & Services: “Force Multiplier”
Team A
Services Scripts/Tools
ENV A
Services Scripts/Tools
ENV B
Services Scripts/Tools
ENV C
Team B Team C
Operations as a Service Platform
Ops
Team n…
Services Scripts/Tools
ENV n

64. Fortune 100 Manufacturing & Services: “Force Multiplier”
Team A
Services Scripts/Tools
ENV A
Services Scripts/Tools
ENV B
Services Scripts/Tools
ENV C
Team B Team C
Operations as a Service Platform
Ops
Team n…
Services Scripts/Tools
ENV n
• Manufacturing and service conglomerate with need to decouple IT

65. Fortune 100 Manufacturing & Services: “Force Multiplier”
Team A
Services Scripts/Tools
ENV A
Services Scripts/Tools
ENV B
Services Scripts/Tools
ENV C
Team B Team C
Operations as a Service Platform
Ops
Team n…
Services Scripts/Tools
ENV n
• Manufacturing and service conglomerate with need to decouple IT
• Each line of business has different security and policy needs

66. Fortune 100 Manufacturing & Services: “Force Multiplier”
Team A
Services Scripts/Tools
ENV A
Services Scripts/Tools
ENV B
Services Scripts/Tools
ENV C
Team B Team C
Operations as a Service Platform
Ops
Team n…
Services Scripts/Tools
ENV n
• Manufacturing and service conglomerate with need to decouple IT
• Each line of business has different security and policy needs
• Handle 5-10x the scale without adding to central Ops organization

67. Recap
Capacity crunch is only
going to get worse
Make explicit investment in
process and tooling
Operations as a Service: Reshaping IT Operations to Solve Today’s Challenges 4
D
evOps and Digital Transformations are
driving an unprecedented increase in
the pace and volume of daily change.
Who generally finds this to be welcome news?
Development and Product teams. Who has reasons
to be alarmed at the problems and challenges this
might bring? Operations.
Operations organizations in today’s enterprises
are finding themselves squeezed between two
unrelenting forces. On one side there are the
business-driven demands of DevOps and Digital
Transformation (“Go faster! Open things up!). On
the other side there are the demands to maximize
security and stability (“Don’t be the next hack! Don’t
be the next outage! Lock things down!”). And there, in
the middle, is an already over-burdened Operations
organization doing their best to avoid being squeezed
beyond the breaking point.
Operations has reached an inflection point. To deliver
what the business demands, Operations must find
a way to provide increasing levels of organizational
responsiveness and throughput — all while “locking
things down” to sufficiently meet today’s risk profiles.
A lot is riding on how Operations responds to this
challenge. A failure here is not just a localized IT
failure. A failure will undermine a business’s ability
to operate. Failing to solve this will turn into a
competitive disadvantage for the business.
On the flip side, this challenge also presents a great
opportunity. Operations can take this business
mandate and use it to reimagine how both planned
and unplanned work is handled. This is a chance to
improve how Operations both serves the broader
business and improves the day-to-day lives of
Operations professionals.
The Operations Squeeze
Introduction
“The Operations Squeeze”
Go faster! Be flexible! Lock it down!
Improved Quality
Shorter Time-to-Market
Fast Feedback
From Users
Availability Auditing
Security Compliance
Dev Ops
Ops
Use a “Lean lens” to
analyze Ops activity
Team A
(Dev)
Team B
(Ops)
Ticket
System
??
Beware of ticket-driven
request queues
Leverage the Operations as
a Service design pattern
#! ! "# $
Scripts APIs Tools Cloud VMs Containers
Orchestration
Collect and
Process Output
Infrastructure
details
Config.
Man.
CMDB
Monitor.
Metrics
Cloud
Corp
Directory
Authentication
ITSM Tickets
>_
● Collaborate on Standard Procedures
● Manage Access & Governance policies
● Execute Actions & Share Visibility
Web GUI API CLI
Operations as a Service Platform
“Shift-Left” control and
decision making.

68. Let’s talk…
@damonedwards
damon@rundeck.com
https://www.rundeck.com/oaas
Deep-dive write-up on Operations as a Service: