Presentation given at Wemanity on March 14th, 2018

2. KEEPING TRACK OF YOUR CI/CD MESS
@DamienCoraboeuf

4. WHO AM I?
▸ Damien Coraboeuf (@DamienCoraboeuf)
▸ Born to Java in 1996
▸ Working in Continuous Delivery since 2008
▸ Automating pipelines at scale

5. ▸ Where the mess comes from
▸ Starting to put some order
▸ From an integrated solution…
▸ … to a central point of knowledge
▸ Extracting customised information
▸ Operational aspects

6. 15000 JOBS IN > 4 YEARS
0
2000
4000
6000
8000
10000
12000
14000
16000
7/15/15 10/23/15 1/31/16 5/10/16 8/18/16 11/26/16 3/6/17 6/14/17 9/22/17 12/31/17
Jobs

7. IT STARTS HERE
Product A
Product B
Project C
Maintenance 11.8
Maintenance 11.9
Maintenance 11.10
Maintenance 7.2
Maintenance 8.0
Maintenance 5.0
Check
Publication
WAS JBoss …
Promotion for QA
QAWAS QAJBoss …
Promotion for NFT
Sonar Fortify Perf
Promotion for delivery
Delivery
Auto Manual
Pipeline
Products & Projects Branches
…

8. RESPONSIBILITIES
Jenkins team
Pipeline libraries
Develops and
maintains
Development team
Seed files
Defines
Seed plug-in
SCM
Triggers

9. PIPELINE AS CODE - META INFORMATION
seed.properties

10. IT’S POURING INFORMATION ALL DAY LONG
A
rtefacts
Depen
d
en
cies
Triggering other
pipelinesRuns
tests
Collects
results
Scans
co
d
e
M
easures
perform
ances
Tags
/
com
m
its
Tickets
Deploying

11. ALL THE TIME

13. EXTRACTING SOME BASIC ORDER
▸ Projects
▸ Branches
▸ Builds
▸ Validations
▸ Promotions
Metadata!!

14. SEEING IS BELIEVING
▸ One tool to rule them all
▸ http://nemerosa.github.io/ontrack/
Promotions
Dependencies
Validations & their statuses
Builds

15. SEEING IS BELIEVING
▸ Filtering
Promoted … despite anomalies in
security scans
▸ Status of your pipeline in one glance

16. CONTRIBUTIONS

17. BUILD INFORMATION

18. SEEING IS BELIEVING
▸ Overview of all your branches

19. SEEING IS BELIEVING
▸ Change logs between builds, between versions
▸ Permalink for automated release notes
▸ Supports GitHub, JIRA, Subversion, BitBucket…

20. SEEING IS BELIEVING
▸ Using cross referenced information for searches:
▸ Where is my ticket?
▸ Where is my commit?
Example: where is fixed by #545 ticket?
Here you go!

21. SEARCH ON BUILDS
Change log

22. SINGLE POINT OF KNOWLEDGE
Ontrack
Artifactory
GitLab
InfluxDB
Jenkins
Vault
JIRA
BitBucket
Subversion
GitHub
Git

23. Ontrack
INTEGRATION WITH THE REST OF THE ECOSYSTEM
Jenkins plug-inGUI
REST API
GraphQL
InfluxDB metrics
Groovy DSL

24. JENKINS PLUG-IN
Post build actions
and
steps

25. JENKINS PLUG-IN
Integration with the Ontrack DSL

26. ONTRACK DSL
▸ Groovy based DSL
Automation of
Change log

27. JOB DSL INTEGRATION
Seed plugin
Job DSL Job Ontrack

28. JENKINS PIPELINE INTEGRATION

29. AUTOMATED FEEDBACK
▸ How to get the best out of your information

30. AUTOMATED FEEDBACK
▸ The collected information can be used in the ecosystem
▸ Based on Ontrack DSL
or GraphQL
▸ Example: promotions used
as triggers in Jenkins

31. AUTOMATED FEEDBACK
▸ Artifact automatic cleanup based on promotions
Collects builds to be deleted
(GraphQL)
Delete artefacts & versions
Cleanup job

32. META INFORMATION AT ALL LEVELS
▸ Attaching built-in or
custom properties to
all levels - projects,
branches, builds, etc.
▸ Used for reporting or
to drive processes
▸ Can be entered
manually or
automatically

33. CUSTOMISING THE SOLUTION
Ontrack Core
Extension
Ontrack
Java API
Extension
code

34. COLLECT YOUR OWN CUSTOM INFORMATION
Ontrack
Foreman
vSphere
HP Fortify
SonarQube
▸ Extension system
▸ New connectors
▸ New services, screens and interfaces

35. CUSTOMISING THE SOLUTION
▸ Deployment catalog
▸ TV wallboards for development teams
▸ Collecting artifact information
▸ Indexing project meta informations
▸ Dependency matrices
▸ KPI dashboards
▸ …
▸ Imagination is the limit

36. CUSTOMISING THE SOLUTION
▸ Deployment catalog
Sends deployment information
(version, platform, date, …)
Deployment job
Overview of deployments
per project

37. CUSTOMISING THE SOLUTION
▸ TV wallboards for development teams
GraphQL / REST

38. CUSTOMISING THE SOLUTION
▸ Collecting artifact information Artifactory
Details of usage / project
Metrics

39. CUSTOMISING THE SOLUTION
▸ Indexing project meta informations
▸ Gradle files, pom.xml, “pipeline as code” definitions
▸ Links with other Ontrack information
▸ Dashboards, search engine, etc.
▸ Extraction of statistics and usage
▸ Complex queries pipeline.type = 'MPL'
and pipeline.major = 3
and !seed['jenkins.pipeline.project.jdk']
and !seed['jenkins.pipeline.sonar.goals']?.contains('toolchains')
and lastBuild.daysSince < 30

40. CUSTOMISING THE SOLUTION
▸ Links between projects at build level collected at build time
▸ Used to display dependency matrices enriched by promotions

41. CUSTOMISING THE SOLUTION
▸ KPI dashboards
Computing high level data for project managers
▸ Aggregating indexed information into high level of knowledge
▸ Computing trends

42. ONTRACK MANAGEMENT
▸ Jobs
▸ Metrics
▸ Deployment
▸ Security model

43. SYSTEM JOBS
▸ ~ 1000 jobs running in the background
▸ Critical system
▸ Collecting information all day long
▸ System administration interface

44. METRICS
▸ Ontrack can export lot of information to InfluxDB
▸ Additional metrics can be created through extensions

45. GETTING STARTED
▸ As simple as:
▸ http://localhost:8080 (admin/admin)
▸ Docker image but also…
▸ Distributions using Debian & RPM packages
▸ Standalone Spring Boot JAR
▸ 2 GB RAM / 2+ CPU / 10 GB Disk
▸ H2 database (local or server)
▸ … but beta version using Postgres
docker container run -d -p 8080:8080 nemerosa/ontrack:2.34.4

46. SECURITY
▸ Authentication
▸ Built-in users
▸ LDAP integration
▸ Extensible
▸ Authorisations
▸ Built-in ACL (roles & functions)
▸ Extensible
▸ Strong isolation between projects
▸ … but cross-projects roles

47. THE FUTURE OF ONTRACK
▸ More integrations: GitLab, Concourse, etc.
▸ Event based (CQRS)
▸ Neo4J export or internal representation

48. ONTRACK
▸ http://nemerosa.github.io/ontrack/
▸ Use it, fill tickets, extend, contribute

49. THANK YOU!
Thank you: Contact:
▸ http://nemerosa.com
▸ @DamienCoraboeuf