This document summarizes a presentation given by Wiebe de Roos and Stefan Simenon of ABN AMRO bank on their transformation to CI/CD practices to accelerate software delivery. It discusses the challenges ABN AMRO previously faced with long lead times, quality issues, and inefficient processes. It outlines their approach to establish prerequisites like tooling and infrastructure, implement CI/CD pipelines, and change management efforts to shift mindsets. Results included improved code quality, deployment frequency, collaboration, and time to market. It advocates for management support, reducing technical debt, creating a safe environment, and focusing on small, continuous improvements over long-term planning.

1. ABN AMRO Transforms with CI/CD to Accelerate
Software Delivery and Improve Security
• DevOps.com webinar
• 27th Mar 2018
• Stefan Simenon/Wiebe de Roos

2. Wiebe de Roos
CI/CD Consultant
• Studied Communication & Multimedia Design
and Master of Management & ICT
• 12+ years of IT expertise
• CI/CD Consultant / Engineer implementing
Jenkins Enterprise in AWS at ABN AMRO
• Speaker at ABN AMRO and other industry
conferences
Speaker Introductions
Stefan Simenon
Head of IT Tooling
& Software Development
• Studied Physics & Information Technology
• 20+ years IT experience
• Responsible for Tooling, Software Quality
& CI/CD at ABN AMRO
• Conference Speaker: Eg Jenkins World,
XebiaLabs Leadership Summit, AllDayDevOps,
Software Quality conferences

3. ABN AMRO is a leading bank with an operating
income of EUR 8588 million
22,000 employees servicing retail, private and
corporate finances worldwide
Headquartered in Amsterdam
5,000 associates working in IT
300+ agile teams
ABN AMRO Overview

4. Many manual handovers and approvals
Long lead time for software delivery
Software quality issues found at a late stage
Code merging happening at a late stage
Inefficient cooperation between Dev and Ops
Big non-frequent releases to production
ABN AMRO Challenges

5. Financial Services Market Conditions

6. Waterfall
Full Agile EnterpriseTraditional Enterprise,
Agile Teams
CI/CD / DevOps Full DevOps Enterprise
Agile & DevOps Transition at ABN AMRO

7. Produce automated builds
and detect errors as soon as
possible, by integrating and
testing all changes on a
regular (daily) basis.
High frequency delivery of a
tested functional piece of
software that can be
deployed to production
rapidly.
Fully automated process
including deployment to
production without human
interaction.
Continuous Integration Continuous DeploymentContinuous Delivery
Need for Faster Response to Clients is Clear

8. CI/CD Program: Set-Up
Extend
Technologies
Move to ET
Automated
Production
Release
Mature in
UT/ST
Start CI/CD in
UT/ST
PAVE THE WAY
Tooling
Infrastructure Prerequisites
Integration
Pipelines
MAKE IT HAPPEN
Change Management
Mindset & Behaviour
Simplify Processes
Coaching for Agile Teams
Front End,
Java

9. CI/CD Program: Approach
CI/CD = A Changed Way of Working and Process Improvements
• Tooling + Mainly Mindset & Behaviour
Organisation = Cluster with a Central and Decentralized Orientation
• Pave the Way: Set Up the Conditions for Teams to Get Working
• Make it Happen: The Actual ‘Decentral’ CI/CD Implementation Within Teams
Agile Teams Supported Once the Right Tools are Available
• Start with Java/Front End
Strong Alignment Across DEV, OPS and SECURITY Departments
Large Companies May Need 3 - 8 Years, Change Approach Along the Way
Keep Overall Stages in Mind
• Plan for the Coming Three Months
• Focus on Learning and Improving vs. of Long-Term Planning

10. Pave the Way – Results so Far (1)
• All Tools Required for Continuous Integration Implemented and Rolled Out
• Various Continuous Integration Pipelines Defines and Implemented
• Pipelines and Integrations Continuously Improved and Extended
• JIRA Agile Toolset Defined and Implemented
• Standard Way of Working Defined and Roll Out in Progress
• From 2,000 to 10,000+ Users in 2 to 5 Years
• Tooling for Release and Deployment Management Selected:
• XL Release and XL Deploy
• Release & Deployment Management WoW Defined and Roll Out in Progress
• Standard CD pipeline for Java/WebSphere, Open Banking and IIB Delivered and Connected to
Standard CI Pipeline
• VSTS Selected and Implemented for Applications Based on Microsoft
• > 100 Applications Onboarded for Automated Deployments
• > 500 XL Release Users

11. Pave the Way – Results so Far (2)
• SonarQube for Code Quality
• HPE Fortify for Secure Coding
• Nexus Life Cycle for OSS Library Management
• Governance to Manage Software Quality Setup, Roll Out in Progress
• Build Breakers Defined, Roll Out in Progress
• Tools Implemented to Enable Automated Testing
• Test Service Virtualization Rolled Out
• Automated Test Data Management, Governance Implemented, Roll Out in Progress
• Automated Test framework Defined and Implemented
• Mainframe Tools Upgraded to Latest Versions
• Identified Strategy to Clean Unused Components and Activities to Recompile Programs
Based on Latest Cobol Compiler 6.1 in order to improve Memory Usage and decrease
MSU Usage
• Mainframe Pipeline Based on Compuware TOPAZ, ISPW, Jenkins and SonarQube in
Progress

12. Midrange Build & Delivery Pipeline: Orchestration
Acceptance
environment (ET)
Production
environment (PRD)
Test environment
(ST)
Zero touch platforms
Deployment
Build
Static
secure
code
Package
Develop
Source
code
Build &
Unit
Tests
Code
quality
scans
Continuous
Integration
Build artefacts
Continuous Delivery
Test data mgmt
ATAF
Test suites
Release Management

13. Tooling
Java
Front End
BPM/
TIBCO
Microsoft
Siebel
Power
Centre/
ETL
IIB
Mainframe
CoTS
Mobile
Pipelines within ABN AMRO

14. Dependency
Scan
Standard CI Pipelines within ABN AMRO and Build Breakers
Check Out
Project from
SCM
Developer
Triggers Build
Build Project and
Execute Unit
Tests
Code Quality
Scan
Secure Coding
Scan
Publish
Deployable
Artifact
N
Y

15. Build Breaker Criteria and Governance
• Senior Management Commitment
• Software Quality Governance
• If Software Quality Criteria are not Met:
• Build will fail and software developer needs to fix/improve the software before being able to publish a
deployable artifact
• Software Quality Criteria and Roll Out of Build Breakers are Defined by:
• Development community consisting of central quality teams
• Representatives in agile teams
• Our application development partners and security department
• Initial Build Breakers in Place for Software Quality, Secure Coding and Dependency Management, Build
Breakers Criteria Strengthened
• Leads to Improved Software, Less Exception Discussions, Improved Mindset.

16. IT4IT Organisation Set Up to Enable CI/CD Implementation
JIRA Dedication
Team
Software
Logistics Team
Application
Deployment
Support Team
Test Tooling
Team
Application
Monitoring Team
Change &
Configuration
Management Team
Portfolio
Management
Team
Application
Logging Team
Implement Tooling Upgrades
Implement New Tools
Enhance and Improve CI/CD Pipelines
Implement New CI/CD Pipelines
Handle User Management
Support Agile Teams
Conduct Incident & Problem Management
Mainframe
Modernization

17. Jenkins: Current environment
Jenkins AnalyticsJenkins Operations
Centre
Master 3
Master X
Master 1
Master 2
Slave
(shared)
Slave
(dynamic)
• 1 Analytics
• 1 JOC
• 10 Masters
• 80+ Slaves
• 128 Connectors
• Supported Enterprise Edition
• 16,000 Jobs
• Approx. 250 Plugins
• 300+ Agile Teams
• 1,500 Developers
• Currently Hosted on Internal
Infrastructure
70+ (!!!) VMs in Datacentre…and GROWING

18. Required Change to Further Scale
• Centralized Maintenance
• Manual Scaling
• Conflict in Tools and Configuration
• Limited Team Autonomy
• Decentralized Maintenance
• Automated Scaling
• Reduced Conflicts in Tools and Configuration
• Increased Team Autonomy
Static VMs Docker Containers
From To

19. Cloudbees Jenkins Enterprise - Architecture
AWS Virtual
Private Cloud
On-Prem
Datacenter
CI
Master Slave
CD
CI
On-Prem
Datacenter

20. Architecture model – functional perspective
Agile teams can maintain their own Jenkins master and run their own pipelines.
This solution prevents interference of teams with each other, reducing conflicts.

21. Cloudbees Jenkins Enterprise – Results & Next Steps
PoC Successfully Conducted
• Successful Installation
• Automated Provisioning of Jenkins Masters
• Usage of Docker Containers
• Communication with Tools on On-Prem Data Centre
• Successful Performance Tests
• Security Issues Identified and Resolved
Cloud Approval Obtained
Risk Assessment Approved
Achieved
• Set Up DTAP CJE Environment
• Connect CJE to ABN AMRO LDAP
• Onboard First Agile Teams
• Migrate from CJE (Mesos/Marathon) to
CJE 2.0 (Kubernetes)
• Establish Standard Pipeline on CJE
• Complete CJE Roll Out in 2019Q1
In Progress/Next Steps

22. 1.
Automate All Repetitive
Tasks
2.
Integrate Quickly and
Often
3.
Everyone is Equally
Responsible
4.
Keep Changes
Small
5.
Get Continuous
Feedback
ABN AMRO CI/CD Key Principles

23. Make It Happen – Results so Far
• CI/CD Summer Event Held
• CI/CD leadership program
• Demos
• Best Practice Sharing
• Training
• Change Management Program Set Up, Focus on Mindset & Behaviour
• CI/CD Coaching Framework Defined, Rollout in Progress
• 100+ Bootcamps Organised and Teams Coached
• Framework Based on Set Deliverables and Team Needs
• CI/CD E-learning Module Delivered and Rolled Out
• Various Communities Set Up
• Internal MeetUps and Hackathons Held
• External Speakers and Tooling Suppliers
• Jez Humble, Josh Long, CloudBees, Sonatype, XebiaLabs, SonarSource
• Platform Set Up - Teams Present Successes, Failures and Best Practices

24. Test Environment Uptime Improved
Improved Code Quality & Secure Coding
Improved Cooperation Across Stakeholders
Improved Time to Market
Improved Development Processes
Realised Benefits within ABN AMRO
Source
code mgt
Build
& Unit test
Code
quality
review
PackageDevelop
Compo-
nent mgt
Deploy Release tests
(ET)
Deploy
Continuous integration
Continuous delivery
Continuous deployment
Prod
checks
Deploy Test (ST)
Zero touch platforms
Code push flow Deployment flowBuild, QA and package flow
x3 Deployments to UT x2,5 Deployments to ET+20% Successful Builds -100% Package Creation Time -75% Testing Time
We never thought it would
be possible to develop, test
and deploy something
completely in one sprint
I-Markets doubled velocity
after 1 sprint containing
CICD improvements only
From 4 Internet
Banking releases to 18
releases per year
Core review times have
been shortened and
violations when
merging are being
prevented
Changes are being
rolled out as soon as
they are available
Increased Velocity
Private Banking
Interlnational team reduced
build from 5 hours to 5
minutes
First continuous deployment
realised by identity access
mgmt team
Release times halved for
teams using XL Release

25. Take Aways
Senior Management Commitment & Involvement
Invest in Reducing Technical Debt
Create a Safe Environment (failing is ok)
Do not Focus on Tooling Only
Do not Underestimate the Journey and Complexity
Do not Focus on Long Term but Rather Small Improvements

26. Database Automation
Automate and Improve Tooling Pipelines
Hybrid Cloud Strategy
Further Transform to DevOps
Improve WoW and Mindset & Behaviour
Facilitate Increased Team Autonomy
Way Forward
CI/CD Metrics

27. 27
Questions