Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
DEVOPS WITH SMELL
ANTONS KRANGA
~ WHOAMI
▸ Full stack developer ~ 15years
▸ Cloud Architect
▸ DevOps evangelist
▸ Innovation Center of Accenture Cloud
Pla...
WHY DEVOPS
DEVELOPERS VERSION
DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
DRIVERS FOR SOFTWARE DEVELOPMENT
▸ development price
BEFORE AFTER
▸ innovations speed
▸ specialization silos
▸ ops comes fi...
DEVOPS IS AN ENVIRONMENT
WHERE PEOPLE TALK
me :)
DEFINITION OF DEVOPS
SMELL SYMPTOMA
DEVOPS ANTIPATTERNS
ANTIPATTERN # 0
UNICORNS VS HORSES
You cannot be a
unicorn when all
others are horses
ANTIPATTERN # 0.1
DEVOPS IN A BOX
You cannot buy culture!
▸ Buy a Golden DevOps LVL5
▸ Become certified DevOps master
▸ Giv...
ANTIPATTERN # 0.2
REBRANDING
DevOps != Configuration Management

DevOps != Release Management|

DevOps != Product Managemen...
ANTIPATTERN # 1
DEVOPS AS A BLACK BOX
What is the value in DevOps?
! Developer
! Sys-Op
ANTIPATTERN # 2
SORRY, NOT MY DEPARTMENT
Bread ownership and
specialization with
autonomous teams
vs
http://martinfowler.c...
ANTIPATTERN # 2.1
DEVOPS AS A SILO
DevOps teams build their own Silo
▸ You cannot talk to DevOps
▸ Use Jira instead !!!!
▸...
ANTIPATTERN # 3
DEFINITION OF DONE
▸ DoD fails with “ilities”
▸ Services can be easily “undone”
ANTIPATTERN # 4
FEAR OF RELEASE
RELEASE === RISK
NO RELEAE NO REVENUE
vs
ANTIPATTERN # 5
SNOWFLAKE SERVER
Applying changes to Server Instance
manually leads to unique and distinct
server configura...
VITAMINS
DEVOPS ANTIPATTERNS
VITAMINS
INFRASTRUCTURE AS CODE
ENV
ENV
ENV
DEV …CODE
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is a

Contract
OPS
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is a

Contract
OPS
Put infrastructure into SCM
VITAMINS
INFRASTRUCTURE AS CODE
DEV
Code is a

Contract
OPS`
Make infrastructure part of app baseline
VITAMINS
WE LIKE CI/CD
▸ We need feedback not to be afraid
▸ Feedback != SPAM
▸ It’s about size of release not frequency
▸...
PAINKILLERS
DEVOPS ANTIPATTERNS
ANTIPATTERN # 6
“JENKINS” DRIVEN DEVELOPMENT
`
`
`
Pipelines are easily becomes your
single point of failure
CI
ANTIPATTERN # 6: PROPOSED SOLUTION
SHIFTING RESPONSIBILITY
CI users (DEVs or OPs) are best for managing their jobs
▸ Use D...
ANTIPATTERN # 7
SLOW PIPELINES
` ` `
Over-engineered Pipelines
Pipeline execution takes too long
Leads to delayed feedback
ANTIPATTERN # 6: PROPOSED SOLUTION
SLOW PIPELINES
`
`
`
Parallelize where you can!
ANTIPATTERN # 7
MANUAL PROMOTION
Engineer PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
ANTIPATTERN # 7: PROPOSED SOLUTION
KILL SWITCH FOR MANUAL TESTING
PRODUATIntegr TestsDEV System Tests
ENVENVENV ENV
TEST
ANTIPATTERN # 7: PROPOSED SOLUTION
AB TESTING
PROD A
Integr TestsDEV System Tests
ENV
ENVENV
TEST A
PROD B
ENV
TEST B
feed...
ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr Tests System Tests
ENVENVENV ENV
DEV B
feature 

branch DEV C
code 

review
ANTIPATTERN # 7.1: PROPOSED SOLUTION
REACTIVE CODEREVIEW
ReleaseIntegr Tests System Tests
ENVENV ENV
Full 

Regression
Cal...
ANTIPATTERN # 7.1
CODEREVIEW
PRODUATIntegr TestsDEV A System Tests
ENVENVENV ENV
master
DEV B
feature 

branch DEV C
code ...
ANTIPATTERN # 8
DEV DEPLOYMENT VIA CI
DEV

ENV
DEV

ENV
DEV

ENV
…DEV CI
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
DEV …NO CI
DEV

ENV
DEV

ENV
DEV

ENV
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
ENV
ENV
DEV

ENV
…DEV NO CI
WANT TO HACK? SURE!
ANTIPATTERN # 8: PROPOSED SOLUTION
DEV DEPLOYMENT VIA CI
ENV
ENV
DEV

ENV
…
YOU BROKE IT? YOU FIX IT!
DEV NO CI
ANTIBIOTICS
DEVOPS ANTIPATTERNS
ANTIPATTERN # 9
GOLDEN IMAGE
VM
OS
Problems
▸ Maintained manually
▸ No collaboration
▸ Hard to distribute
▸ Non versioning...
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
Chnorr Service
ANTIPATTERN # 9: PROPOSED SOLUTION
PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
cmdb
Chnorr Servi...
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
▸ Provisioning success < 100...
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
▸ Provisioning success < 100...
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
▸ Provisioning success < 100...
ANTIPATTERN # A
FRAGILE PROVISIONING
OS
Configure
Infrastructure

code
Harden Download Install
▸ Provisioning success < 100...
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
...
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
...
ANTIPATTERN # A.1
PRIVATE DATA CENTER PROVISIONING
PRIVATE DATA CENTER: NO INTERNET!!!
OS
ConfigureHarden Download Install
...
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
static dynamic
OS
ConfigureHarden Download Install
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
Chnorr Service
ANTIPATTERN # A.1: PROPOSED SOLUTION
STEM CELLS
OS
ConfigureHarden Download Install
LAB PRIVATE DATA CENTER
code
packer
PRO...
ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructure

code
Harden Download Install
▸ Version of libraries
▸ Version ...
ANTIPATTERN # B
DEPENDENCY HELL
OS
Configure
Infrastructure

code
Harden Download Install
▸ Version of libraries
▸ Version ...
ANTIPATTERN # B: PROPOSED SOLUTION
CANARY BUILDS
Accept Tests
CI
Unit Test Provision
…
PRECISE version libs
Accept Tests
C...
ANTIPATTERN # C
INFRASTRUCTURE PET
Attributes of Pet
▸ Have meaningful names
▸ Long living instance
▸ Often needs manual n...
ANTIPATTERN # C: PROPOSED SOLUTION
INFRASTRUCTURE CATTLE
Attributes of Pet
▸ Have numbers in its name
▸ Short living insta...
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
ANTIPATTERN # C: PROPOSED SOLUTION
PET TO TRANSFORMATION CATTLE EXAMPLE
Chnorr Service
Database
Cattle
Pet
User Data
ANTIPATTERN # D
SECRETS LEAK
OS
Configure
Infrastructure

code
Install
cmdb
wrong place 

for your secrets
wrong place 

fo...
ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Don’t store secrets with code
▸ Don’t store secrets with configuration
▸ ...
ANTIPATTERN # D: PROPOSED SOLUTION
SECRETS LEAK
▸ Secret can be leased and rotated
▸ tmpfs is your fiend
Secrets DOs
ANTIPATTERN # D: PROPOSED SOLUTION
SECURITY LEASING EXAMPLE
Chnorr Service vault
consul
IAM
Database
api
x hours leasing
A...
TAKEAWAYS
GOOD INFRA CODE
TAKEAWAYS
LEARN PATTERNS BEFORE TOOLS
Patterns Tools
vs
TAKEAWAYS
EVERYTHING MUST HAVE AN API
REST
DSL
CLI
TAKEAWAYS
SELF TESTABLE CODE
▸ Use assertions for infrastructure code
▸ Use acceptance test frameworks
Tests improves your...
TAKEAWAYS
READING
▸ Book: A Human Error Approach to
Aviation Accident Analysis
▸ Author: Douglas A. Wiegmann

Scott A. Sha...
TAKEAWAYS
READING
▸ Book: Clean Code
▸ Author: Robert C Martin
▸ ISBN: 978-0132350884
THANK YOU
Upcoming SlideShare
Loading in …5
×

DevOps With Smell

Presented at DevTernity 2015 (www.devternity.com)

It appeared hard to do DevOps just right. This talk will be focused around experience gathered doing on-field consulting of multiple DevOps projects. We will focus on both pillars of DevOps: engineering and processes and go through the pitfalls, anti-patterns and some healthy critics of DevOps.

  • Be the first to comment

DevOps With Smell

  1. 1. DEVOPS WITH SMELL ANTONS KRANGA
  2. 2. ~ WHOAMI ▸ Full stack developer ~ 15years ▸ Cloud Architect ▸ DevOps evangelist ▸ Innovation Center of Accenture Cloud Platform ▸ Speaker ▸ Marathon runner ANTONS KRANGA
  3. 3. WHY DEVOPS DEVELOPERS VERSION
  4. 4. DRIVERS FOR SOFTWARE DEVELOPMENT ▸ development price BEFORE AFTER ▸ innovations speed
  5. 5. DRIVERS FOR SOFTWARE DEVELOPMENT ▸ development price BEFORE AFTER ▸ innovations speed ▸ specialization silos ▸ ops comes first ▸ cross functional ▸ software defined
 data centers ▸ complex communication ▸ complexity theory
  6. 6. DEVOPS IS AN ENVIRONMENT WHERE PEOPLE TALK me :) DEFINITION OF DEVOPS
  7. 7. SMELL SYMPTOMA DEVOPS ANTIPATTERNS
  8. 8. ANTIPATTERN # 0 UNICORNS VS HORSES You cannot be a unicorn when all others are horses
  9. 9. ANTIPATTERN # 0.1 DEVOPS IN A BOX You cannot buy culture! ▸ Buy a Golden DevOps LVL5 ▸ Become certified DevOps master ▸ Give me DevOps compliance checklist ▸ Give me 5 key DevOps control metrics
  10. 10. ANTIPATTERN # 0.2 REBRANDING DevOps != Configuration Management
 DevOps != Release Management|
 DevOps != Product Management
 …
 DevOps != (*) Management DEVOPS IS THE CULTURE YOU CANNOT FIND IN IN ITIL CATALOG
  11. 11. ANTIPATTERN # 1 DEVOPS AS A BLACK BOX What is the value in DevOps? ! Developer ! Sys-Op
  12. 12. ANTIPATTERN # 2 SORRY, NOT MY DEPARTMENT Bread ownership and specialization with autonomous teams vs http://martinfowler.com/bliki/DevOpsCulture.html
  13. 13. ANTIPATTERN # 2.1 DEVOPS AS A SILO DevOps teams build their own Silo ▸ You cannot talk to DevOps ▸ Use Jira instead !!!! ▸ RTFM Architecture
  14. 14. ANTIPATTERN # 3 DEFINITION OF DONE ▸ DoD fails with “ilities” ▸ Services can be easily “undone”
  15. 15. ANTIPATTERN # 4 FEAR OF RELEASE RELEASE === RISK NO RELEAE NO REVENUE vs
  16. 16. ANTIPATTERN # 5 SNOWFLAKE SERVER Applying changes to Server Instance manually leads to unique and distinct server configuration footprint (TECHNICAL DEBT) http://martinfowler.com/bliki/SnowflakeServer.html
  17. 17. VITAMINS DEVOPS ANTIPATTERNS
  18. 18. VITAMINS INFRASTRUCTURE AS CODE ENV ENV ENV DEV …CODE
  19. 19. VITAMINS INFRASTRUCTURE AS CODE DEV Code is a
 Contract OPS
  20. 20. VITAMINS INFRASTRUCTURE AS CODE DEV Code is a
 Contract OPS Put infrastructure into SCM
  21. 21. VITAMINS INFRASTRUCTURE AS CODE DEV Code is a
 Contract OPS` Make infrastructure part of app baseline
  22. 22. VITAMINS WE LIKE CI/CD ▸ We need feedback not to be afraid ▸ Feedback != SPAM ▸ It’s about size of release not frequency ▸ Don’t judge for broken builds ▸ Go home when build is green
  23. 23. PAINKILLERS DEVOPS ANTIPATTERNS
  24. 24. ANTIPATTERN # 6 “JENKINS” DRIVEN DEVELOPMENT ` ` ` Pipelines are easily becomes your single point of failure CI
  25. 25. ANTIPATTERN # 6: PROPOSED SOLUTION SHIFTING RESPONSIBILITY CI users (DEVs or OPs) are best for managing their jobs ▸ Use DSL to build CI/CD pipelines ▸ Store CI/CD pipelines in git 
 if possible make it part of app baseline ▸ Generate and bootsrap pipelines with API
  26. 26. ANTIPATTERN # 7 SLOW PIPELINES ` ` ` Over-engineered Pipelines Pipeline execution takes too long Leads to delayed feedback
  27. 27. ANTIPATTERN # 6: PROPOSED SOLUTION SLOW PIPELINES ` ` ` Parallelize where you can!
  28. 28. ANTIPATTERN # 7 MANUAL PROMOTION Engineer PRODUATIntegr TestsDEV System Tests ENVENVENV ENV
  29. 29. ANTIPATTERN # 7: PROPOSED SOLUTION KILL SWITCH FOR MANUAL TESTING PRODUATIntegr TestsDEV System Tests ENVENVENV ENV TEST
  30. 30. ANTIPATTERN # 7: PROPOSED SOLUTION AB TESTING PROD A Integr TestsDEV System Tests ENV ENVENV TEST A PROD B ENV TEST B feedback feedback measure
  31. 31. ANTIPATTERN # 7.1 CODEREVIEW PRODUATIntegr Tests System Tests ENVENVENV ENV DEV B feature 
 branch DEV C code 
 review
  32. 32. ANTIPATTERN # 7.1: PROPOSED SOLUTION REACTIVE CODEREVIEW ReleaseIntegr Tests System Tests ENVENV ENV Full 
 Regression Calc technical
 debt ENV Nightly DEV A Standup code 
 review …
  33. 33. ANTIPATTERN # 7.1 CODEREVIEW PRODUATIntegr TestsDEV A System Tests ENVENVENV ENV master DEV B feature 
 branch DEV C code 
 review
  34. 34. ANTIPATTERN # 8 DEV DEPLOYMENT VIA CI DEV
 ENV DEV
 ENV DEV
 ENV …DEV CI
  35. 35. ANTIPATTERN # 8: PROPOSED SOLUTION DEV DEPLOYMENT VIA CI DEV …NO CI DEV
 ENV DEV
 ENV DEV
 ENV
  36. 36. ANTIPATTERN # 8: PROPOSED SOLUTION DEV DEPLOYMENT VIA CI ENV ENV DEV
 ENV …DEV NO CI WANT TO HACK? SURE!
  37. 37. ANTIPATTERN # 8: PROPOSED SOLUTION DEV DEPLOYMENT VIA CI ENV ENV DEV
 ENV … YOU BROKE IT? YOU FIX IT! DEV NO CI
  38. 38. ANTIBIOTICS DEVOPS ANTIPATTERNS
  39. 39. ANTIPATTERN # 9 GOLDEN IMAGE VM OS Problems ▸ Maintained manually ▸ No collaboration ▸ Hard to distribute ▸ Non versioning Chnorr Service
  40. 40. ANTIPATTERN # 9: PROPOSED SOLUTION PROVISIONING OS Chnorr Service
  41. 41. ANTIPATTERN # 9: PROPOSED SOLUTION PROVISIONING OS Configure Infrastructure
 code Harden Download Install Chnorr Service
  42. 42. ANTIPATTERN # 9: PROPOSED SOLUTION PROVISIONING OS Configure Infrastructure
 code Harden Download Install cmdb Chnorr Service
  43. 43. ANTIPATTERN # A FRAGILE PROVISIONING OS Configure Infrastructure
 code Harden Download Install ▸ Provisioning success < 100% ▸ Time to provision Chnorr Service
  44. 44. ANTIPATTERN # A FRAGILE PROVISIONING OS Configure Infrastructure
 code Harden Download Install ▸ Provisioning success < 100% ▸ Time to provision Chnorr Service
  45. 45. ANTIPATTERN # A FRAGILE PROVISIONING OS Configure Infrastructure
 code Harden Download Install ▸ Provisioning success < 100% ▸ Time to provision Chnorr Service
  46. 46. ANTIPATTERN # A FRAGILE PROVISIONING OS Configure Infrastructure
 code Harden Download Install ▸ Provisioning success < 100% ▸ Time to provision Chnorr Service
  47. 47. ANTIPATTERN # A.1 PRIVATE DATA CENTER PROVISIONING PRIVATE DATA CENTER: NO INTERNET!!! OS ConfigureHarden Download Install Chnorr Service
  48. 48. ANTIPATTERN # A.1 PRIVATE DATA CENTER PROVISIONING PRIVATE DATA CENTER: NO INTERNET!!! OS ConfigureHarden Download Install … apt yum gem Chnorr Service
  49. 49. ANTIPATTERN # A.1 PRIVATE DATA CENTER PROVISIONING PRIVATE DATA CENTER: NO INTERNET!!! OS ConfigureHarden Download Install … apt yum gem complicated? Chnorr Service
  50. 50. ANTIPATTERN # A.1: PROPOSED SOLUTION STEM CELLS static dynamic OS ConfigureHarden Download Install Chnorr Service
  51. 51. ANTIPATTERN # A.1: PROPOSED SOLUTION STEM CELLS OS ConfigureHarden Download Install LAB PRIVATE DATA CENTER Chnorr Service
  52. 52. ANTIPATTERN # A.1: PROPOSED SOLUTION STEM CELLS OS ConfigureHarden Download Install LAB PRIVATE DATA CENTER code packer PROVISIONSNAPSHOT docker Chnorr Service
  53. 53. ANTIPATTERN # B DEPENDENCY HELL OS Configure Infrastructure
 code Harden Download Install ▸ Version of libraries ▸ Version of packages ▸ Maintain dependencies Chnorr Service
  54. 54. ANTIPATTERN # B DEPENDENCY HELL OS Configure Infrastructure
 code Harden Download Install ▸ Version of libraries ▸ Version of packages ▸ Maintain dependencies ▸ Version of your infra code ▸ Maintain dependencies Chnorr Service
  55. 55. ANTIPATTERN # B: PROPOSED SOLUTION CANARY BUILDS Accept Tests CI Unit Test Provision … PRECISE version libs Accept Tests CI Unit Test Provision … LATEST version libs Chnorr Service Chnorr Service
  56. 56. ANTIPATTERN # C INFRASTRUCTURE PET Attributes of Pet ▸ Have meaningful names ▸ Long living instance ▸ Often needs manual nursing ▸ Requires scary patching ▸ Leads to snowflakes ▸ PaaS is modern pet
  57. 57. ANTIPATTERN # C: PROPOSED SOLUTION INFRASTRUCTURE CATTLE Attributes of Pet ▸ Have numbers in its name ▸ Short living instance ▸ Immutable configuration ▸ Recreate instead of patching ▸ Requires careful planning
  58. 58. ANTIPATTERN # C: PROPOSED SOLUTION PET TO TRANSFORMATION CATTLE EXAMPLE Chnorr Service Database
  59. 59. ANTIPATTERN # C: PROPOSED SOLUTION PET TO TRANSFORMATION CATTLE EXAMPLE Chnorr Service Database
  60. 60. ANTIPATTERN # C: PROPOSED SOLUTION PET TO TRANSFORMATION CATTLE EXAMPLE Chnorr Service Database Cattle Pet
  61. 61. ANTIPATTERN # C: PROPOSED SOLUTION PET TO TRANSFORMATION CATTLE EXAMPLE Chnorr Service Database Cattle Pet User Data
  62. 62. ANTIPATTERN # D SECRETS LEAK OS Configure Infrastructure
 code Install cmdb wrong place 
 for your secrets wrong place 
 for your secrets Chnorr Service
  63. 63. ANTIPATTERN # D: PROPOSED SOLUTION SECRETS LEAK ▸ Don’t store secrets with code ▸ Don’t store secrets with configuration ▸ Don’t leave secrets in service Secrets DON’Ts
  64. 64. ANTIPATTERN # D: PROPOSED SOLUTION SECRETS LEAK ▸ Secret can be leased and rotated ▸ tmpfs is your fiend Secrets DOs
  65. 65. ANTIPATTERN # D: PROPOSED SOLUTION SECURITY LEASING EXAMPLE Chnorr Service vault consul IAM Database api x hours leasing AWS
  66. 66. TAKEAWAYS GOOD INFRA CODE
  67. 67. TAKEAWAYS LEARN PATTERNS BEFORE TOOLS Patterns Tools vs
  68. 68. TAKEAWAYS EVERYTHING MUST HAVE AN API REST DSL CLI
  69. 69. TAKEAWAYS SELF TESTABLE CODE ▸ Use assertions for infrastructure code ▸ Use acceptance test frameworks Tests improves your confidence
  70. 70. TAKEAWAYS READING ▸ Book: A Human Error Approach to Aviation Accident Analysis ▸ Author: Douglas A. Wiegmann
 Scott A. Shappell ▸ ISBN: 978-0754618737
  71. 71. TAKEAWAYS READING ▸ Book: Clean Code ▸ Author: Robert C Martin ▸ ISBN: 978-0132350884
  72. 72. THANK YOU

×