2. DTO Solutions
Friday, October 7, 2011 2
We do process improvement and automated infrastructure for companies who build and
operate revenue producing services... e-commerce, financial services, gaming
3. Are you an ...
aaS
Friday, October 7, 2011 3
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
4. Are you an ...
[spi] aaS
Friday, October 7, 2011 3
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
5. Are you an ...
[spi] aaS
Friday, October 7, 2011 3
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
6. Are you an ...
[spi] aaS
Friday, October 7, 2011 3
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
7. Are you an ...
[spi] aaS
Friday, October 7, 2011 3
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
8. Are you an ...
[spi] aaS
Friday, October 7, 2011 3
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
9. Are you an ...
[spi] aaS
Friday, October 7, 2011 3
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
10. Are you an ...
[spi] aaS
Friday, October 7, 2011 3
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
11. Friday, October 7, 2011 4
-How many ppl are familiar with the metaphor used in softw dev called “TD”?
-In softw dev you typally have two choices get it done quick and take hit of future issues or
spend to time on a cleaner design, but will take longer to put in place.
-Quick typically p with a technical debt, which is similar to a financial debt.interest payments,
which come in the form of the extra effort that we have to do in future development because
of the quick and dirty design choice.
-
-Most of you are probably familiar with “Software” TD
-I am going to tell you a story about “Infrastructure TD”
12. Friday, October 7, 2011 5
-How many ppl know who these guys are?
-The dudes who invented Facebook.. rght?
-However, this was not there first venture.
-Green vs Red Widgits compnay story.
-After many arguments the both decide to split up. Cameron to make green widgets and Tyler
decides to make red widgets.
-They both go to their father each ask for a 1m dollars and the father asks how much money
are you going to make. They both say 10 million.
-Father calculates the ROR to be 900% and gives em each 1m.
10-1 = 9/1 = 900%
-However one of them lied.... The red widgets only return 233%
10-1-2 = 7/3 = 233%
13. Green vs Red
Widgits
1 Million 1 Million
2 Million No TD
10 Million Profit 10 Million Profit
10 - 3 = 7/3 10 - 1 = 9/1
233% ROR 900% ROR
Friday, October 7, 2011 6
14. It Gets Worse...
Technical Vicious Toxic
Terminal
Debt Cycle Operations
Friday, October 7, 2011 7
- It gets even worse. Isreal Ghat of the Cutter Group call the vicious cycle of TD.
- You wind up fixing a lot of things that you didn’t fix in the first place..
- this pulls more resources from delivery good service and compound effect is that you are spending more and more
resources that you should have gotten right the firs time..
- but ever worse the effect of customer satisfaction starts loosing more business and the “V” cycle is out of control.
- TD->VTD->Toxic operations->Terminal
- I call this are you running a business or building a business?
- Toxic operations. Amertrade/etrade story
15. Tale of Two Startups
Friday, October 7, 2011 8
- Jesse Robbins my ex boss and CEO of Opscode/Chef did a great post on O’rielly rdar a few years ago called the Tale of
Two Startups.
- The chart looked like this first 4 weeks.
- First chart legacy (I call it the non devops startup/project)
- Second Chart is the (secret sauce startup ... #devops)
- I played around with this using “R” to be cool and I came up with a=140% ROR and 700% ROR
16. The Meat to Math Ratio
http://radar.oreilly.com/2011/08/meat-to-math-ratio.html
Friday, October 7, 2011 9
- Alistar Croll has a great post called the meat to math ratio.
- Amazon had $12.95B in Q410 revenues and 33,700 employees, revenue per employee of $384,273.
- For Barnes & Noble: Barnes & Noble: $1.91B in Q410 revenues, and 35,000 employees, meaning a revenue per employee
of $54,571.
- Netflix: $444M in Q409 revenues, and 1,000 employees, meaning a revenue-per-employee of $444,000.
- Blockbuster: $400M in Q409 revenues. The company peaked at 60,000 employees i
- Dropbox: In Q211 Dropbox had $25M in revenues, and 74 employees, for a revenue per employee of $338K.
17. When do they call?
Friday, October 7, 2011 10
We started asking ourselves some questions about our clients and our work for them
18. When do they call?
A. First signs of issues brewing
B. Initial negative impact felt by some
C. Heads are on fire
Friday, October 7, 2011 10
We started asking ourselves some questions about our clients and our work for them
19. When do they call?
10% A. First signs of issues brewing
30% B. Initial negative impact felt by some
65% C. Heads are on fire
Friday, October 7, 2011 10
We started asking ourselves some questions about our clients and our work for them
20. Why did they wait?
#1 Answer:
Friday, October 7, 2011 11
Aside from human nature to procrastinate?
translation... “you and or the problem you are telling me about isn’t worth the money”
21. Why did they wait?
#1 Answer:
“Couldn’t get budget approval
or business support”
Friday, October 7, 2011 11
Aside from human nature to procrastinate?
translation... “you and or the problem you are telling me about isn’t worth the money”
23. Operations has a perception problem
Business View
Necessary Cost
Friday, October 7, 2011 12
24. Operations has a perception problem
Business View Ops View
vs
Necessary Cost Strategic Weapon
Friday, October 7, 2011 12
25. First Law of Business Spending
Things that Things that
cost you money make you money
Friday, October 7, 2011 13
Make no mistake about it... someone is spending money on you... and they are asking
themselves... do I cut it, outsource it, or spend more on it.
26. Change the Perception of Ops
vs
Necessary Cost Strategic Weapon
Friday, October 7, 2011 14
27. Operations becomes a strategic weapon
Friday, October 7, 2011 15
If we are going to make the business case that operations is a strategic weapon... and we
need to do it in language that they understand.
Business care about innovation (ideas that resonnate with customers) and getting the best
return on their money... Let’s look at the first point... continuously increasing velocity of
innovation
28. Operations becomes a strategic weapon
When you are continuously...
Friday, October 7, 2011 15
If we are going to make the business case that operations is a strategic weapon... and we
need to do it in language that they understand.
Business care about innovation (ideas that resonnate with customers) and getting the best
return on their money... Let’s look at the first point... continuously increasing velocity of
innovation
29. Operations becomes a strategic weapon
When you are continuously...
1. Increasing velocity of innovation
Friday, October 7, 2011 15
If we are going to make the business case that operations is a strategic weapon... and we
need to do it in language that they understand.
Business care about innovation (ideas that resonnate with customers) and getting the best
return on their money... Let’s look at the first point... continuously increasing velocity of
innovation
30. Operations becomes a strategic weapon
When you are continuously...
1. Increasing velocity of innovation
2. Increasing return on investment
Friday, October 7, 2011 15
If we are going to make the business case that operations is a strategic weapon... and we
need to do it in language that they understand.
Business care about innovation (ideas that resonnate with customers) and getting the best
return on their money... Let’s look at the first point... continuously increasing velocity of
innovation
31. Result
Ah-ha!
Friday, October 7, 2011 16
Core process of any business
32. Ah-ha! Ka-ching!
Friday, October 7, 2011 17
Now of course you hope that result is one that resonates with customers
34. Velocity of Innovation...
Ah-ha! Ka-ching!
Day 0 Day n
Friday, October 7, 2011 19
That cycle time... getting from idea to result is one of the most critical metrics for both a
startup and an established service. And the bulk of this time is spent in the application
lifecycle across dev, QA, and operations.
35. Companies were able to
achieve somewhat defensible
positions based on
technology...
Friday, October 7, 2011 20
36. Companies were able to
achieve somewhat defensible
positions based on
technology... then came the
web
Friday, October 7, 2011 21
Then along came this thing called the web and screwed it all up... now your customers are
coming to you through a standard interface --the browser-- over standard published
protocols. You competition is only a few keystrokes away. The even applies in newer
distribution channels like mobile apps and their app stores.
37. How do we compete now?
Friday, October 7, 2011 22
So how do we compete in this commoditized world?
1. Scale... scale of users, scale of data, scale of compute power
2. Velocity of innovation... how fast can you react to and execute on new market forces or
opportunities.
Scaling is the most straight forward problem to solve. It’s a known problem with a lot of
known solutions to borrow from. Hire smart architects... let them follow best practices...
throw a big pile of cash at it and there aren’t many scaling problems you can’t solve.
Obviously that’s somewhat of a simplification... but I’ll let someone else’s presentation get
into the details
38. How do we compete now?
1. Scale
Friday, October 7, 2011 22
So how do we compete in this commoditized world?
1. Scale... scale of users, scale of data, scale of compute power
2. Velocity of innovation... how fast can you react to and execute on new market forces or
opportunities.
Scaling is the most straight forward problem to solve. It’s a known problem with a lot of
known solutions to borrow from. Hire smart architects... let them follow best practices...
throw a big pile of cash at it and there aren’t many scaling problems you can’t solve.
Obviously that’s somewhat of a simplification... but I’ll let someone else’s presentation get
into the details
39. How do we compete now?
1. Scale
2. Velocity of Innovation
Friday, October 7, 2011 22
So how do we compete in this commoditized world?
1. Scale... scale of users, scale of data, scale of compute power
2. Velocity of innovation... how fast can you react to and execute on new market forces or
opportunities.
Scaling is the most straight forward problem to solve. It’s a known problem with a lot of
known solutions to borrow from. Hire smart architects... let them follow best practices...
throw a big pile of cash at it and there aren’t many scaling problems you can’t solve.
Obviously that’s somewhat of a simplification... but I’ll let someone else’s presentation get
into the details
40. How do we compete now?
1. Scale
2. Velocity of Innovation
Friday, October 7, 2011 22
So how do we compete in this commoditized world?
1. Scale... scale of users, scale of data, scale of compute power
2. Velocity of innovation... how fast can you react to and execute on new market forces or
opportunities.
Scaling is the most straight forward problem to solve. It’s a known problem with a lot of
known solutions to borrow from. Hire smart architects... let them follow best practices...
throw a big pile of cash at it and there aren’t many scaling problems you can’t solve.
Obviously that’s somewhat of a simplification... but I’ll let someone else’s presentation get
into the details
41. Innovation is really a numbers game...
Ah-ha! Ka-ching!
Ah-ha!
Friday, October 7, 2011 23
The global innovation success rate across all geographies and industries
42. Innovation is really a numbers game...
4%*
Ah-ha! Ka-ching!
94%*
Ah-ha!
*Study by Doblin Innovation Consultants
Friday, October 7, 2011 23
The global innovation success rate across all geographies and industries
43. Innovation is really a numbers game...
4%*
Ah-ha! Ka-ching!
94%*
Ah-ha!
*Study by Doblin Innovation Consultants
Friday, October 7, 2011 23
The global innovation success rate across all geographies and industries
44. How to win a numbers game...
Company A
Result
Ah-ha!
Company B
Result Result Result Result
Ah-ha! Ah-ha! Ah-ha! Ah-ha!
Friday, October 7, 2011 24
Put it this way... if in the time it takes you to get through one cycle, your competitor can get
through 4... who do you think will be more competitive??
45. How to win a numbers game...
Company A
Result
Ah-ha!
Company B
Result Result Result Result
Ah-ha! Ah-ha! Ah-ha! Ah-ha!
Friday, October 7, 2011 25
This is why the notion of a “burn rate” is a meaningless statistic. Say you have enough cash in
the bank to pay for x number of months. That doesn’t tell you anything... do you have 10
shots at making customers happy or just 1? Think about it like a carnival game. If you have
ten balls to throw at the target and someone else only has 1... who has a better probability of
success?
46. How fast can ops move?
• Production deployment every 11.6
seconds (weekday)
• 1,079 deployments in one hour (record)
• ~0.001% of deployments actually cause an
outage
http://news.ycombinator.com/item?id=2971521
Friday, October 7, 2011 26
Jon Jenkins at Velocity
Ops is amazon’s strategic weapon... their books aren’t better, their storage isn’t better, their
hypervisor isn’t better.
Everyone copies them but their just keep rolling out features and lowering costs faster than
their competitors can do either
47. Business is already thinking about this...
Customer Development Model
Friday, October 7, 2011 27
But can the biz move fast enough? They already are wanting to do so
Customer Development
49. Operations becomes a strategic weapon
When you are continuously...
Friday, October 7, 2011 28
50. Operations becomes a strategic weapon
When you are continuously...
1. Increasing velocity of innovation
Friday, October 7, 2011 28
51. Operations becomes a strategic weapon
When you are continuously...
1. Increasing velocity of innovation
2. Improving return on investment
Friday, October 7, 2011 28
52. Friday, October 7, 2011 29
•I got into a Twttier argument that went like this...
• See “Cloud Gone Wrong”
53. Clouds Gone Wild
Busine
s s
Dev
AWS
business s3put s3
ssh
ideas !
apache/php Rightscale
memcached
mysql
Friday, October 7, 2011 30
-So letʼs start the story in the begining... Biz guy got a great idea for a service.
-They used a classic web2.0 app architecture: apache/php, memcached & mysql.
Development done on a single server
-Production ran on EC2 and they used rightscale server templates
-Release done by pushing code and assets to s3 buckets and then running a parallel SSH
scripts to distribute them
-This approach seemed to work... They got up to a few hundred nodes pretty fast.. business
was cooking
54. apache/php
p arty,
3rd s3
memcached
& are AWS mysql
w
mi ddle
Rightscale
yum
Business
apache/php puppet
memcached
mysql apache/php
s3
AWS s3 memcached
AWS mysql
Rightscale
Rightscale
s3put s3put
ssh ssh
Business2 Business3
Friday, October 7, 2011 31
-First service was such a huge success they decided to launch other sevices
-So they ”Copy and pasted” the whole architecture and lifecycle to launch the new businesses
-Each new group pushed assets to s3, scripted the distribution, and hacked the rightscripts and
templates
-Things were obviously getting more complicated, so they did what they were supposed to do
and added centralized tooling like puppet and yum
-They thought they were doing things the cloud way and that all would be fine
55. “As-is”
operations server
templates
TAR
RPM restarts
TAR
TAR
code,
EC2
content pupp
deploys et RS
app RS deployment
devs deployment
S3
yum
repo
Provision-time builds
reconfigs
TAR EXE
TAR AUTO
server
TOOLS
code templates
TAR
RPM
middleware
devs platform CONTROL PROVISION RELEASE
system
eng
Friday, October 7, 2011 32
First we got the team on the whiteboard to map out the “as is” picture. This is a, believe it or
not, a simplified version of that.
Some of the highlights...
-First youʼll notice that different groups had their own path to production... different methods of
control, provisioning, and release.
-Each group and role seemed to have a different way to editing or storing config
-There were differing ways of packaging software... sometimes it might be a .tar.gz other times
it might be an RPM.
-Shockingly... Some things were even being built directly on production servers.
-There was no authoritative source of information is maintained about nodes, application
topology, software versions, etc....
57. Infrastructure Deveopment Life Cycle
Everything starts
here
SVN
hud agent cfg
yum pup
son repo pet
packages
active users run packages
directory deck
commands
nag splunk
ios
resource model
sys cfg
right instance create
scale
events
new node Node a log data
CONTROL PROVISION RELEASE
Friday, October 7, 2011 34
I wonʼt go into too much detail about the tooling that was put into place to support all of this but
here are some highlights..
-took a loosely coupled toolchain approach... using mostly open source tools
-This became their standard stack of “operations middleware”. Of course, we are all used to
the notion of application middleware... but to an online service, the management infrastructure
is just as much a part of the service as the application itself.
-This operations middleware stack is a first class citizen along with application stack and it all
goes through the same SDLC... everything is versioned, built, deployed, and packaged via the
same process
-Once in place, this middleware provides a single path for releasing, provisioning, and
controlling anything that goes into an environment.
59. DevOps Cafe
subscribe in
iTunes
Friday, October 7, 2011 36
I also do a podcast with the famous cloud and IT management guru john willis.
Interview based series where we talk to all kinds of movers and shakers across the
development and operation spectrum.
Between DTO and doing the devops cafe content I get to talk to a lot of companies and see
what’s working and what isn’t working.
60. Let’s Talk....
@botchagalupe
john@dtosolutions.com
Friday, October 7, 2011 37
We do this stuff all day long for a lot of large and cutting edge clients... and we love talking
about DevOps so drop me a line anytime if you want to talk
61. Let’s Talk....
@botchagalupe dev2ops.org
john@dtosolutions.com
Friday, October 7, 2011 37
We do this stuff all day long for a lot of large and cutting edge clients... and we love talking
about DevOps so drop me a line anytime if you want to talk
63. C
A
M
S
Friday, October 7, 2011 39
First get you mind around what you are looking for... Simple framework for categorizing
DevOps problems and solutions.
64. Culture
Automation
Measurement
Sharing
Friday, October 7, 2011 40