About the idea of DevOps, why we implemented DevOps and what we did, what is important ! About our road from waterfall/ITIL and silo structures to DevOps/Agile culture.

1. Forum Speed IT, Warsaw 19/11/2015
Jerzy Gulczyński, Dreamlab.pl
DevOps
By Dreamlab

4. IS DEVOPS LIKE TEENAGE SEX…?
www.inzynierem.pl
Everyone
talks about it
No one really knows
how to do it
Everyone thinks
Everyone else is doing it
So everyone claims they are doing it.

5. REALITY BASED ON PUPPET LABS 2014
REPORT
„Our analysis makes it clear that DevOps teams are
a growing trend.”
16% of respondents were part
of a DevOps department
https://puppetlabs.com/2014-devops-report
57% are implementing, or have
already implemented DevOps
practices

6. WHAT IS DEVOPS
ALL IN A SELF-ORGANIZED TEAM AND ONE CYCLE
AGILE
EXTENSION
CULTURE OF CONTINUOUS IMPROVEMENT

7. DEVOPS CALMS
• Focus on People
• Embrace Change & ExperimentationCULTURE
• „Continuous Delivery”
• „Infrastructure as Code”Automation
• Focus on producing value for end user
• Small batch sizesLean
• Measure everything
• Show the improvementMeasurement
• Open information sharing
• Collaboration & CommunicationSharing

8. WHY #1
OF
BUSSINES
Puppets Labs findings:
OF BUSINESS

9. OF
BUSSINES
OF BUSINESS
„High-performing IT organizations
• experience 60 times fewer failures
• and recover from failure [MTTR] 168
times faster than their lower-
performing peers
• They also deploy 30 times more
frequently
with 200 times shorter lead times”
https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf
WHY #1

10. WHY #2
JOB SATISFACTION
HAPPY COWS MAKE BETTER CHEESE

11. IT WAS YEAR 2006
To organise, make it „professional”
and the results predictible

12. ROAD TO MATURITY
NETWORK
1stLINE
UNIX
DBA
FRONTEND
PM’s
MIDDLEWARE
DATABASE
DEVELOPMENT OPERATIONS

13. UPTIME
NETWORK
1stLINE
UNIX
DBA
FRONTEND
PM’s
MIDDLEWARE
DATABASE
DEVELOPMENT OPERATIONS
TIMECOST
SCOPE
ROAD TO MATURITY

14. LESSONS LEARNED
CREATIVITY ?
FORGET
JOB SATISFACTION ?
BURN OUTI CAN RUN FASTER
SPEED LIMIT 60 KM/h ?

15. ROAD TO AGILE
In 2011 we went into AGILE

16. And into DEVOPS in 2012
ROAD TO AGILE

17. Development Operations
2nd line1st lineFrondend DatabasesMiddleware
NET UNIX DBA
SILO STRUCTURE, PROJECT ORIENTED
Project Management Office
Quality Assurance Team

18. Development Operations
2nd line1st line
NET UNIX DBA
Project Management Office
Quality Assurance Team
STEP #1 PRODUCT ORIENTED (NOT PROJECT]
Product 1 Product...Product 2
SG VOD
Product 2
Zumi …

19. Development Operations
2nd line1st lineProduct 1 Product...Product 2 Product 2
BETTER, BUT…
I want a
change
I want
stability
Who is
responsible
Accountability++ but limited due to lack of competences
Number of changes++ but still controlled and slowed down by ITIL
24 support in a product oriented teams getting hurt
Inconsistent architecture due to separate product teams
Fuzzy responsibility from business point of view

20. STEP #2 PLATFORMS
PaaS
Product
s
Queue
CMS
OCDN
Search
Big Data Video
RDBS
Cache
NO SQL
Object St.
…
…
Vod.pl
Onet.pl
Wiadomosci.onet.pl
….pl / …rs / …hu / …cz

21. Architecture Board
Continuous Delivery Team
Master of Service Team
Products Platforms
Cloud
PaaS
Services
VideoSG VOD
…
STEP #3 DEVOPS TEAMS
Bigdata …
Project Management Office
Quality Assurance Team [provides QA standards and tools]
Solution Product Owners
Email
Zumi
Social CMS
Monitoring as a service Team

22. PRODUCT TEAM (NOT PROJECT)
They Make It,
They Break It,
They Fix It.

23. WHEN PRODUCT IS TOO BIG - DIVIDE IT VERTICALLY
Video Platform
Team
VOD Service
Team

24. Agile & DevOps

25. CULTURE OF LEARNING
Purpose
Mastery
Autonomy
Trust
Self-organization

26. HIGH ALIGNMENT AND AUTONOMY
http://blog.crisp.se/author/henrikkniberg

27. CLEAR GOALS AND WIDER PERSPECTIVE
Company
goals
Quarterly goals
for Teams
DevOps
Demo days
Quarterly
Loop

28. Demo Days
SHARE YOUR KNOWLEDGE, BRAG, CELEBRATE

29. ENGAGE THE PEOPLE AND LET THEM CREATE COMPANY CULTURE
HackDay

30. ENGAGE THE PEOPLE
BarCamp

31. BLAMELESS POSTMORTEMS
WHAT IF I TOLD YOU
IT WAS A BUG IN A THIRD-PARTY SYSTEM
• Do not blame for failures
• Be focused on conclusions
and on actions after
• Be transparent to Business

32. TEAM HEALTH CHECK & IMPROVEMENT
BOARDS

33. MEASURE EVERYTHING
• Log everything
• Provide logs and events open
for all devops
we collect 130k
logs, events per
second
1 mln metrics
in Graphite
Online console, Graphite, Hadoop

34. Monitoring is intended for administrators only.
People who don't know how to create it and
thereby who are not able to interpret it are kindly
REQUESTED NOT to run alarms and NOT to disturb
administrators.
MONITORING THE PAST

35. Monitoring Center and MaaS Team
REACTIVE MONITORING

36. PROACTIVE MONITORING
Monitoring TVs in Teams’ Rooms

37. MONITORING FEEDBACK LOOP
MaaS Team [24/7/365]
Product/TeamReactive
Proactive
Monitoring points (buttons)
Troubleshooters/HowTo’s
Escalation Path
Trainings
First aid based on Howto's
Fake alarms, lack of
alarms

38. DEPLOY OFTEN
10+ deployments / day
50-60 deployments / day
Every 3 minutes (in average 200 daily)
Every 11.6 seconds
Deployments rate

39. CONTINUOUS DEPLOYMENT
Automated and boring process
Development
Unit Tests,
Lint Score
Code
Review
Integration
Tests
Functional
Tests
Deployment
Monitoring

40. AB TESTING
WHEN
DEPLOYING
AB TESTING WHEN DEPLOYING
Deploy on 10% of traffic, check, increase

41. PET
Pets

42. Cattle

43. INFRASTRUCTURE AS CODE
orchestrator
PaaS
orchestrator
IaaS
servers
VMapplications

44. TO BE INFRASTRUCTURE INDEPENDENT
Amazon
EC2
Amazon S3
Onet DataCenter Credit Card
Amazon ELB
Business
continuity plan
4h a day
1% of traffic

45. OWN PRIVATE VS PUBLIC
IS IT STILL POSSIBLE TO CHASE
PUBLIC CLOUD SERVICES ?
UNFORTUNATELLY NOT,
NO LONGER
Amazon AWS with over 1 million active customers
S3 data transfer has increased by 120%.Y2Y
Databases services increased by 127% Y2Y
EC2 usage increased by 95% Y2Y

46. PRIVATE CLOUDS ARE NO LONGER ABLE TO COMPETE WITH PUBLIC
„Private cloud will not keep up with the public cloud
dynamics of innovations, breadth and depth of offering.”
„80% of private cloud fails today as no one wants to use it:
developers prefer public cloud”
„Cloud is a software platform game not a hardware
commodity game.”
„Yes, you can decide to use IaaS in most commoditized
mode, build your orchestration and middleware layers, and
now you are locked into your own in-house technology and
you will maintain it for the rest of your career.”
Gartner:

47. Capital One: "We want to build great applications for our
customers, not to invest in costly & complex infrastructure
[...] we believe it enables us to operate more securely in a
public cloud that we can even could in our own datacenters”
General Electric „The Cloud has gone from probable to
inevitable. We intend to move over 60% of our global
workload into AWS. For us this is no longer an experiment,
it’s no longer a test, it’s not something we talk being
probable, its inevitable.”
ENTERPRISES ARE REALLY USING AMAZON

48. BIMODAL IT ?

49. DON’T JUMP INTO DEVOPS IN A WATERFALL
WAY

50. LINKS
One piece flow: https://www.youtube.com/watch?v=Dr67i5SdXiM
Puppet Labs Reports:
https://puppetlabs.com/2014-devops-report
https://puppetlabs.com/sites/default/files/2015-state-of-devops-report.pdf
Microservices: http://martinfowler.com/articles/microservices.html
Small Batch: http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-
improve-flow/

51. BOOKS