Our DevOps Journey Transforming 6 Month Waterfalls to 1 Hour Code Deploys https://info.dynatrace.com/17q3_wc_from_agile_to_cloudy_devops_na_registration.html In the 2nd part of our webinar series, Anita Engleder, DevOps Lead at Dynatrace reviews and dissects lessons learned during the transformational journey moving Dynatrace from an on-prem culture to one that is cloud native. She will lend her perspective as a key member of the team that executed on the original vision: to “implement a new cloud native offering and deploy a new feature release every 2 weeks. Additionally, be able to support a 1-hour lead time from Code Change to Production”. On November 17th at 1pm/10am PT Anita will present the challenges she and her team faced transforming 6 Months Waterfall to 1 Hour Code Deploys. In this webinar Anita will discuss: How to enable a complete cultural shift across multiple teams, in terms of thought process AND execution What the specific role of her DevOps team is and how it played into the transformation The role of Feature teams and why continuous feedback is critical for them How to successfully influence key stakeholders for complete alignment Today Anita’s team runs 170 production changes every day, running across several AWS Data Centers as well as On-Premise – something that would have been thought impossible only a few years prior.

1. Our DevOps Journey
Transforming 6 Month Waterfalls to 1 Hour Code Deploy
http://dynatrace.com/trial
Anita Engleder
DevOps Manager

2. From the DevOps Webinar with Gene & Mark
Mark Tomlinson
Performance Sherpa
@mark_on_task
Andi Grabner
Performance Advocate
@grabnerandi
Gene Kim, CTO
Researcher and Author
@RealGeneKim

3. @RealGeneKim
24 “Features in a Box” Ship the whole box!
Very late feedback 
@grabnerandi

4. @RealGeneKim
Continuous Innovation and Optimization
1 “Feature at a Time”
Optimize Before DeployImmediate Customer Feedback
@grabnerandi

5. From 0 to DevOps in 80 days
Lessons learnt from shifting an on-prem to a cloud culture
Bernd Greifeneder, CTO
http://dynatrace.com/trial
Webinar: http://ow.ly/cEYo305kFEy
Podcast: http://bit.ly/pureperf

6. 2 major releases/year
customers deploy & operate on-prem
26 major releases/year
170 prod deployments/day
self-service online sales
SaaS &
Managed
2011 2016

7. 7 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
believe in the mission impossible
6months
major/minor release
+ intermediate fix-packs
+ weeks to months
rollout delay
sprint releases (continuous-delivery)
1h : code to production

8. Dynatrace Transformation by the numbers
23x
170
more releases
Deployments / Day
31000 60hUnit+Int Tests / hour UI Tests per Build
More Quality
~200 340code commits / day Stories per sprint
More Agile
93%
Production bugs found by Dev
@grabnerandi
More Stability
450 99.998%Global EC2 Instances Global Availability

9. DevOps Stories & Practices Learned
•The Role of the DevOps Team
•The Role of the Feature Team
•The Journey to DevOps
•The DevOps Tool Chain
@grabnerandi

10. Typical Role of DevOps Teams
mainly responsible for …
… monitoring Production and creating tickets for engineers
… building deployment automation

11. NOT mainly responsible for …
Role of Dynatrace DevOps Team
… monitoring Production and creating tickets for engineers
… building deployment automation

12. owning and promoting tooling that gives engineering teams …
Role of Dynatrace DevOps Team
… VISIBILITY & CONTINUOUS FEEDBACK
from commit to deployment and into all deployments
… CONTROL to deploy features/fixes
… OWNERSHIP of their own source code for all deployments

13. acting as
Engineers
Role of Dynatrace DevOps Team
Dynatrace Managed/SaaS
Orchestration Layer
DynatracePipeline Visualization
Deployment Timeline
Log Overview
using Dynatrace Log APIJIRA Integrations
&
Product Managers

14. confidential
Role of Dynatrace
Feature Teams

15. ManualCICDFcontinuous
integration
continuous delivery & deployment
and feedback

16. Role of Dynatrace Feature Teams
DevOps
=
Engineers start thinking
like an Ops
before Commit
CDF
develop
features &
provide fixes
build & adjust
test coverage
implement
go-to-
production/
migration
plan
add/use proper
feedback
channel
consume
feedback
improve &
harden
CI

17. confidential
DevOps
Our Journey to
CDF & NoOps

18. Our Journey to CDF & NoOps
1
3
2
4
5
6
7
Infrastructure/Platform as a Service
API call instead fixing hardware
Architects are responsible
for production too
Same monitoring in every stage!
Visibility into every stage/deployment!
Same Deployment Automation in
every stage! No manual touches!
Direct Feedback to Engineers!
The real experts are primary consumers
Control into engineer’s hand wherever
it makes sense.
Who pushes the button “deploy”?Provide fast lane to production for
Hotfixes! 1 Hour Code Deploy!
3
4
5
7

19. 3 Same Deployment Automation in every stage! No manual touches!
Dynatrace Managed
Orchestration Layer
Dynatrace SaaS
Orchestration Layer
Orchestration Layer
Product owned by a Feature Team
having it’s own CDF Pipeline

20. 4 Same monitoring in every stage! Visibility into every stage/deployment!
Dev Stage Daily Production StageAcceptance Stage Bi-Daily

21. 4 Same monitoring in every stage! Visibility means also easy access!
https://www.dynatrace.com/meet-davis/

22. 4 Same monitoring in every stage! Visibility means also easy access!
https://www.dynatrace.com/meet-davis/

23. 5 Direct Feedback to Engineers!
Engineers typically like Logs. But how to get the whole context?

24. Thank you!
Food for thoughts:
• How is your trust level to your engineering team?
• Do you encourage failure and learn from it?
• Where do you currently have bottlenecks from Dev to Ops?
• Can technology help? Can process change help?
• Do you have the right monitoring strategy / feedback loop?
http://dynatrace.com/trial

25. http://dynatrace.com/trial
Thank you!

26. 7 Provide fast lane to production for Hotfixes!
1 Hour Code Deploy
How can I achieve that
in case my trunk/master is not yet
at any time production ready?

27. Dev Stage Daily
Continuous Delivery & Feedback
Production Stage
Bi-Weekly & On Demand
Acceptance Stage Bi-Daily
Deploy
Develop &
Fixing
Acceptance &
Performance
& Load Tests &
Monitor
Deploy
Fixing
Acceptance &
Performance
& Load Tests &
Monitor
Deploy
Hotfixing
Monitor
Release
Every 2 weeks version is pushed to next stage = 2 weeks release cycle
Unit+Integration
testing & Build
Unit+Integration
testing & Build
Unit+Integration
testing & Build
Week 1 Week 2 Week 3 Week 4

28. CDF Tooling Chain - Dev Stage
Acceptance Tests
Selenium/Appium Automated Functional Tests
on UI Level
Automated E2E Tests on UI Level
Browser Compatibility Testing
(Chrome, IE, FF, Edge, Safari)
24/7 Deployment Checks
Jira Manual Regression Testing
AWS, VMware, VirtualBox Test Environments
Develop
Gradle + Artifactory build/deployment automation
SVN + Git Version Control
Eclipse / IntelliJ IDEA
Quickbuild + Jenkins Continuous Integration
Unit & Integration Tests (fully automated)
Unit Tests +Integration Tests
Quickbuild, Eclipse/IntelliJ IDEA for local
execution before commit
Memory/CPU Overhead tests
Dynatrace AppMon
Code Quality and Coverage Checks
Sonar
Memory leak detection
Bullseye, Valgrind
Virus Scan Kaspersky
Open Source License compliance checks
Blackduck
Facts/Numbers
28.000 Unit Tests + 3.000 integration test executions / hour
Deployment (fully automated)
Quickbuild Automated daily deployment of Trunk Builds
using Ansible and Puppet or calling
Dynatrace Ruxit CloudControl, that uses
AWS Cloud Formation
(whole infrastructure as code!)
Selenium/Appium
Automated Customer-Like Deployment
Deployment Checks on UI Level
Performance & Load Tests
Cluster Workload Simulator
Eclipse Mem Analyzer (MAT) + Eclipse Thread Dump
Analyzer
Java Flightrecorder
Monitoring
Dynatrace Real-User, Service and
Infrastructure monitoring
WebChecks, AWS Monitoring, Log Analytics
& Monitoring
Continuous Feedback Tools built by Devs & DevOps
OpsGenie and common HipChat Room to
escalate detected Problems directly to
Development
Security 24/7 OSSEC Host intrusion monitoring
Deploy
Develop &
Fixing
Unit+Integration
testing & Build
Acceptance &
Performance
& Load Tests
& Monitor

29. CDF Tooling Chain - Acceptance Stage
Deploy
Fixing
Unit+Integration
testing & Build
Acceptance &
Performance
& Load Tests
& Monitor
Facts/Numbers (Dev + Acceptance Stage)
~700 Automated UI Tests
~60 hours UI Test execution per Build  ~20
parallel running test sets executed on ~30
execution machines up to ~5 hours per test set
~15 different OS (Windows, Linux)
Unit & Integration Tests (fully automated)
Unit Tests +Integration Tests
Quickbuild, Eclipse/IntelliJ IDEA for local
execution before commit
Virus Scan Kaspersky
Deployment (fully automated)
Quickbuild Automated deployment of Sprint Builds
using Ansible and Puppet or calling
Dynatrace Ruxit CloudControl, that uses
AWS Cloud Formation
(whole infrastructure as code!)
Selenium/Appium
Automated Customer-Like Deployment
Deployment Checks on UI Level
Acceptance Tests
Selenium/Appium Automated Functional Tests
on UI Level
Automated E2E Tests on UI Level
Browser Compatibility Testing
(Chrome, IE, FF, Edge, Safari)
24/7 Deployment Checks
Jira Manual Regression Testing
AWS, VMware, VirtualBox Test Environments
Fixing
Gradle + Artifactory build/deployment automation
SVN + Git Version Control
Eclipse / IntelliJ IDEA
Quickbuild + Jenkins Continuous Integration
Performance & Load Tests
Cluster Workload Simulator
Eclipse Mem Analyzer (MAT) + Thread Dump Analyzer
Java Flightrecorder
Monitoring
Dynatrace Real-User, Service and
Infrastructure monitoring
WebChecks, AWS Monitoring, Log Analytics
& Monitoring
Continuous Feedback Tools built by Devs & DevOps
OpsGenie and common HipChat Room to escalate
detected Problems directly to Development
Security 24/7 OSSEC Host intrusion monitoring

30. CDF Tooling Chain - Production Stage
Deploy
Hotfixing
Unit+Integration
testing & Build
Monitor
Fixing
Gradle + Artifactory build/deployment automation
SVN + Git Version Control
Eclipse / IntelliJ IDEA
Quickbuild + Jenkins Continuous Integration
Monitoring
Dynatrace Real-User, Service and
Infrastructure monitoring
WebChecks, AWS Monitoring, Log Analytics
& Monitoring
Continuous Feedback Tools built by Devs & DevOps
OpsGenie and common HipChat Room to escalate
detected Problems directly to Development
Security 24/7 OSSEC Host intrusion monitoring
Monthly and on demand Vulnerability Scans
(KPMG Linz)
Closed Bug Bounty Program at HackerOne
Unit & Integration Tests (fully automated)
Unit Tests +Integration Tests
Quickbuild, Eclipse/IntelliJ IDEA for local
execution before commit
Virus Scan Kaspersky
Deployment (fully automated)
Quickbuild Automated deployment of Sprint Builds
using Ansible and Puppet or calling
Dynatrace Ruxit CloudControl, that uses
AWS Cloud Formation
(whole infrastructure as code!)
Selenium/Appium
Automated Customer-Like Deployment
Deployment Checks on UI Level