A short timelined description of DevOps at Morpho made for ParisDevOps meetup of 2nd of December 2014

1. 0 /
Morpho DevOps approach time line
Jean-Charles JOREL (jean-charles.jorel@morpho.com)
May. 2015
DevOps Infrastructure
Service Line

2. 1 /
DevOps approach brief
 Agenda
 A word about Me
 Time travel from 2010 to 2014

3. 2 /
About me…
 Jean-Charles JOREL (jean-charles.jorel@morpho.com)
 DevOps Service Line Manager
 Leading a Team of 18 people dedicated to DevOps deployment
& associated operations
 Safran Morpho Expert
 Promote Morpho Technical Excellence outside of the
corporation
 Areas of Expertise: DevOps…, Cloud Techs, Network
protocols & SDN, Innovation process, Linux hacking…
 Help to bring new Tech Trends inside Morpho

4. The DevOps@Morpho
time line
(Presentation made for the ParisDevops meetup of 2nd of December 2014)

5. 4 /
Mid-2010 : Root of everything…
 Our Products were using ‘old
school’ technologies mainly
based on C++ and Home made
frameworks
 Our Products were specialized
on a small set of OS and HW
Morpho DevOps Service Line
2010 2013 2014 2015
Morpho creates the
Morpho Core Platform
initiative & associated
Team
MCP
Let’s migrate to support of wide choice
of OS (Linux’es + Windows) and
Open Frameworks with 

6. 5 /
Mid-2012 : Houston, we have a problem…
 So many Operating Systems,
so many App Servers, so
many components, that could
be seen as ‘constantly moving
parts’
 But also, so many subjects
where Experts are not in the
MCP Team…
Morpho DevOps Service Line
2010 2013 2014 2015
Morpho Core Platform has
successes but also difficulties…
(inability to test the full scope at the
right pace, team seen by others as
not legitimate on many subjects)
Time for a choice:
1. Stop or reduce the scope,
2. Change the way we are working to
increase drastically Productivity
Let’s Try Option 2. !

7. 6 /
Mid-2012 : First « DevOps » word usage but…
 Once you have a « Software forge », teams may want to
move further by doing deeper Integration tests
 This quickly leads to issues regarding « software forge » IT
environment not matching the target deployment platform,
overload the forge resources, possible contamination of forge
environment.
 That’s one of the reason why SW teams
always required access to real test
benches where they can do integration
 “DevOps” is a term describing the
seamless link of the 2 needs:
 SW forge to build and do unit testing
 Relevant integration test environments
where to deploy what we just compiled
SVN
SVN
Too
Early!!

8. 7 /
Mid to End-2012: Not focusing on DevOps but Shaping
a PoC about Test Env Automation
Got a High Profile sponsor for a LEAN GreenBelt
Made >40 meetings to explain to all involved teams (Dev &
Ops + IT Dept) expected benefits
 Try to focus on highly visible gains linked to Test Bench
management
15 involved people (PoC Builders & early Users)…
Morpho DevOps Service Line

9. 8 /
-IaaS manager
-VM repository
-Block Storage (SAN)
Virtual IaaS network #nVirtual IaaS network #1
2012/Proposed PoC setup
Server 2012 VM
+ ActiveDirectory
+ OpenAM + NTP
CentOS VM +
PuppetMaster +
DNS + DHCP
CentOS VM +
Supervision
COTS+ PostFix
CentOS VM +
Puppet AgentCentOS VM +
Puppet AgentCentOS VM +
Puppet Agent
REHL6 VM +
Oracle RAC
RHEL6.3 VM +
Oracle RAC
Node
Dashboard
access
Physical
nodes
Server 2012 VM
+ ActiveDirectory
+ OpenAM + NTP
CentOS VM +
PuppetMaster +
DNS + DHCP
CentOS VM +
Supervision
COTS+ PostFix
CentOS VM +
Puppet AgentCentOS VM +
Puppet AgentRHEL 6.3 VM +
Puppet Agent
REHL6 VM +
Oracle RACCentOS VM +
PostGreSQL HA
Nexus +
YUM
Windows 7 VM +
Puppet Agent
Deployment
configuration

10. 9 /
End-2012 to March 2013 : A successful PoC!
Proved that we can automate Test Env Deployment
with the OpenStack IaaS!
 Perfect IP Address Plan segregation confirmed so allowing our
Must-have Datacenter cloning use-case!
Let’s apply the concept to
Morpho DevOps Service Line
2010 2013 2014 2015
MCP

11. 10 /
DevOps Practices & Platform
2013: “DevOps” word only known as a way to help MCP
+ =
DBA, On-Site
installers,
Site support,
Network
engineers,
…
Ex: Java
developper,
integrators,
Testers, … MCP
 We ask Ops to deliver pre-packaged IT to the MCP Team (and so Dev
teams)
 Oracle Database, Operating Systems, Supervision tools, Security hardening

12. 11 /
March 2013 to Oct-2013:
Building the first « DevOps » tool chain
A small set of MCP Team working on it BUT with external
contributions from other teams,
From 10 to 100 users involved (Only volunteers)
First benefits:
 Sharp increase of Test coverage of MCP
 First packaged Oracle deliverables from DBA
 Quick-wins of OpenStack usage (snapshots) by users
Morpho DevOps Service Line
2010 2013 2014 2015
MCP

13. 12 /
Mid-2013 : First DevOps platform concept
Extended scope
MCP
Programs
DevOpsplatform
Product & Suites …/…Components
(Simplified platform view)

14. 13 /
Nov-2013 to March-2014:
Viral usage of « DevOps » tool chain
150 users involved in France, Ireland, Germany and
Morocco
 All from Dev (almost none from Ops side… not yet…)
Creation of the DevOps Service Line on 1st of January!
 DevOps word is officially used as a Practice!!
…but most people still do not know exactly what it is… 
Morpho DevOps Service Line
2010 2013 2014 2015

15. 14 /
Morpho DevOps Infrastructure, Technologies, Process
Supervision &
SLA
OS,
Virtualization &
Networks
Database
Management
Installation &
Lifecycle
Suite
or
Product
or
MSU
Suite
or
Product
or
MSU
Suite
or
Product
or
MSU
Suite
or
Product
Agile
Application
Platform
MCP
Common
Binary Storage
Solutions
Solutions
SolutionsPrograms &
Solutions
Agile
DevOps
Our Devs… Our Ops…

16. 15 /
Morpho DevOps Service Line
4 teams for Full Product to Program Lifecycle support
Stress Test services &
Test Benches operations
Software Configuration
Management
Artifact Lifecycle
Management
Standardized IT
Deliverables
MCP
Collaborative Software
Methods
for developers & integrators

17. 16 /
DevOps Service Line teams
Software
Method definitions
SCM & ALM
Operations
IT pre-packaging for Tools
& Product / Solutions
Stress Test Mgmt for
Products & Solutions
HW & OS
Technology scouting
SCM/ALM Op team
~5 people
Software Collaborative
Method team
~3 people
Test Bench Operations
(OpenStack for Dev/Int/Test)
MCP-IT team
~5 people
Stress Test &
Test Bench team
~5 people

18. 17 /
March-2014 to June-2014:
Hard days for the DevOps Service Line
DevOps platform is near collapse... Still more and
more users but unhappy with the SLA
 New Oracle RAC use-cases had catastrophic impacts on the Platform
 Lots of unexpected Support toward users rediscovering some IT aspects
 Lots of OpenStack Grizzly issues due to heavy Orchestration use-cases
Emergency investments performed!!
 Local storage with SSD for OpenStack Compute Nodes solved most of issues
 Strong automation effort of supervision and administration tasks to govern DevOps
platform resource usage in near real-time
Morpho DevOps Service Line
2010 2012 2013 2014

19. 18 /
July-2014 to now:
First big rewards!
DevOps platform under heavy usage and integrated by most
of Dev teams
 Very interesting early DevOps use-cases shown (see next slide)
 People thinking Automation and Resilience
 Best-Practices for DevOps Platform Resource Governance accepted by users
 Under process to make most our Ops also use the Platform…
 Ok for new Customer Programs, Difficulties with Ops supporting our legacy
(process to move back and forth out/in DevOps platform not yet mature…)
 Envisioning future with more and more DevOps agility (OpenStack over
OpenStack and OpenStack over other clouds…)
Morpho DevOps Service Line
2010 2012 2013 2014

20. 19 /
DEVOPS@MORPHO: FIRST EARLY SUCCESS
STORIES
 Our products stressing continuously their integration with Oracle RAC
 Permanent Performance & Resilience test campaign
 One Oracle RAC failover simulated every ten minutes
Automated through OpenStack API directly by the Dev team. Not possible cheaply before
OpenStack IaaS!
 One critical random bug discovered in JBoss 7.2 leading to data loss
“1 out of roughly 100,000 requests at an Oracle RAC fail over event every 10 minutes would
translate into 1 out of 15 fail over events assuming a throughput of 1 million requests in 24
hours (which was roughly the throughput of the test system). Assuming that the customer has
a bad database which has an Oracle RAC fail over event every month, a data loss will
statistically occur every 15 months of processing”
Bug discovered in Development phase instead of Production!
 Fixed before Product official release! 
DevOps approach provided the ability to forecast impacts of a rare
random issue by compressing time

21. 20 /
DevOps vs non-DevOps: How to identify them based
on their Test Resource consumption ?
 DevOps oriented projects are only possible if you have a Test & Integration
Infrastructure providing a near unlimited amount of very cheap IT resources
Amount of Test & Integration resources used
End of project End of project
DevOps project non-DevOps project
Powered by

22. 21 /
Annex
CONFIDENTIEL / DATE / DIRECTION

23. 22 /
0
200
400
600
800
1000
1200
0
5000
10000
15000
20000
25000NumberofVMscreatedpermonth
janv.-13 févr.-13 mars-13 avr.-13 mai-13 juin-13 juil.-13 août-13 sept.-13 oct.-13 nov.-13 déc.-13 janv.-14 févr.-14 mars-14 avr.-14 mai-14 juin-14
OpenStack Instances created 28 463 618 318 274 676 658 1095 1045 1762 4689 6436 6140 9890 10359 13562 14536 21849
OpenStack Instances deleted 27 413 572 287 234 619 634 1049 979 1715 4641 6442 6005 9814 10252 13409 14453 21678
OpenStack Active instances 4 14 49 76 124 183 224 294 238 401 474 349 626 694 783 617 966 1100
VM resources
KPI for the IAAS
(Infrastructure-as-a-service)
One VM created
every 2 minutes!

24. 23 /
75,0%
80,0%
85,0%
90,0%
95,0%
100,0%
105,0%
0
200
400
600
800
1000
1200
NumberofVMS
Ratios
janv.-13 févr.-13
mars-
13
avr.-13 mai-13 juin-13 juil.-13 août-13 sept.-13 oct.-13 nov.-13 déc.-13 janv.-14 févr.-14
mars-
14
avr.-14 mai-14 juin-14
OpenStack Active instances 4 14 49 76 124 183 224 294 238 401 474 349 626 694 783 617 966 1100
Turnover ratio (global) 85,7% 97,1% 95,6% 94,7% 92,7% 92,3% 92,6% 92,9% 95,4% 94,2% 95,9% 98,1% 97,4% 98,0% 98,2% 98,9% 98,7% 98,8%
Turnover ratio (month) 96,4% 89,2% 92,6% 90,3% 85,4% 91,6% 96,4% 95,8% 93,7% 97,3% 99,0% 100,1% 97,8% 99,2% 99,0% 98,9% 99,4% 99,2%
VM Turnover ratios
KPI for the IAAS More than 1000 Equivalent
Full Time VMs but 99,2%
lived less than one month…

25. 24 /
0,0%
20,0%
40,0%
60,0%
80,0%
100,0%
120,0%
0
500
1000
1500
2000
2500
3000
3500
4000
4500
Capacity RAM (GBytes)
Active RAM (Gbytes)
RAM usage ratio
KPI for the IAAS [RAM selling]
New servers added!
(Stress Test /
Performance HW
zones)
Test benches are
bigger and bigger to
manage more
complex use-cases
(Various Suite and Stress
Test activities)

26. 25 /
0,0%
20,0%
40,0%
60,0%
80,0%
100,0%
120,0%
140,0%
160,0%
180,0%
200,0%
0
100000
200000
300000
400000
500000
600000
700000
800000
vCPU Capacity Hours
VCPU Hours
vCPU usage ratio
KPI for the IaaS [vCPU selling]
Increase of vCPU
hour selling.
New servers added
especially for Stress
Test activities