How can businesses modernize their existing data integration flows? How can they connect a rapidly evolving number of data services? How can they capture, process, and generate new event streams? How can they leverage advances in Machine Learning to enhance real time interactions with their customers? Join Matt Aslett, Research Director at 451 Research, and Jürgen Leschner from Pivotal for an interactive discussion about continuous data integration applications, trends, and architectures. In this webinar you will learn: - How traditional data integration approaches like batch ETL can be improved - Why microservices support continuous data integration in a scalable way - How to incorporate DevOps practices in your data integration teams - What benefits microservices and DevOps practices bring to data integration Presenters: Jurgen Leschner, Pivotal and Matt Aslett, Research Director, 451 Research

1. 1
Microservices &
Continuous Data Integration
Dormain Drewitz - Pivotal
Matt Aslett - 451 Research
Jürgen Leschner - Pivotal

2. 2
Today’s speakers
Panelist
Matt Aslett
Research Director
451 Group
Panelist
Jürgen Leschner
Product Team
Pivotal
Moderator
Dormain Drewitz
Platform Ecosystem
Pivotal

3. 3
Agenda
1. Backdrop: Digital Transformation
2. Data Integration: What’s changed?
3. The Case for Microservices for Continuous
Data Integration
4. Day 2: People and Platforms
5. Looking toward the Future

4. Copyright (C) 2017 451 Research LLC
451 Research is a leading IT research & advisory company
Founded in 2000
300+ employees, including over 120 analysts
2,000+ clients: Technology & Service providers, corporate
advisory, finance, professional services, and IT decision makers
50,000+ IT professionals, business users and consumers in our research
community
Over 52 million data points published each quarter and 4,500+ reports
published each year
3,000+ technology & service providers under coverage
451 Research and its sister company, Uptime Institute, are the two divisions
of The 451 Group
Headquartered in New York City, with offices in London, Boston, San
Francisco, Washington DC, Mexico, Costa Rica, Brazil, Spain, UAE, Russia,
Taiwan, Singapore and Malaysia
Research & Data
Advisory
Events
Go 2 Market

5. 5
Pivotal Labs
Agile Practice Pivotal Data Suite
Greenplum GemFire
HDB
Pivotal Tracker
Organize and Plan
Spring
Build reliably
on Spring Boot
Concourse
Continuous Integration and
Deployment
Pivotal Cloud Foundry
Multi-Cloud Platform
for Cloud-Native Apps

6. Digital Transformation
6

7. Copyright (C) 2017 451 Research LLC
Digital Transformation: What do we mean?
7
Copyright (C) 2017 451 Research LLC
When IT innovation is aligned with and driven by a well-planned business
strategy to:
! transform how organizations serve customers, employees, and partners
! support continuous improvement in business operations
! disrupt existing businesses and markets
! invent new businesses and business models

8. The Case for Microservices
Why Microservices?
1. Increased development velocity
2. Organizational scale
3. Increased agility
4. Faster onboarding
5. Cost efficient scaling
6. Attracting/retaining talent
7. Fault isolation
Continuous Delivery is a precursor to
deliver microservices in a safe manner
- Cassandra Shum, Thoughtworks
What do Microservices need?
1. Agile development
2. Test-driven development
3. Continuous integration
4. Continuous deployment
5. DevOps
6. Distributed Security
7. Intelligent Traffic Management
8. Visibility

9. Data Integration: What’s
changed?
9

10. Copyright (C) 2017 451 Research LLC
Engagement and Intelligence: Retail example
Customer service is driven by systems of
engagement. Traditionally that engagement was
physical, and person-to-person
Help, advice, suggestions and recommendations
were provided and questions were answered (or
not) by the employee
Even the transaction was performed physically,
before being entered into the system of record
(financial/ERP, CRM applications)
That data was then made available for analysis

11. Copyright (C) 2017 451 Research LLC
SYSTEMS OF ENGAGEMENT
Traditional systems of engagement and analysis
SYSTEMS OF ANALYSIS
DATA
ANALYSTS
DECISION
MAKERS
IT PROS
SYSTEMS OF RECORD

12. Copyright (C) 2017 451 Research LLC
SYSTEMS OF ENGAGEMENT
New systems of engagement and analysis
SYSTEMS OF ANALYSISSYSTEMS OF RECORD
DATA
SCIENTISTS
DECISION
MAKERS
BUSINESS
USERS
DATA
ANALYSTS

13. Copyright (C) 2017 451 Research LLC
New systems of engagement require intelligence
SYSTEMS OF ANALYSISSYSTEMS OF RECORD
SYSTEMS OF ENGAGEMENT
DATA
ANALYSTS DATA
SCIENTISTS
DECISION
MAKERS
BUSINESS
USERS
SYSTEMS OF INTELLIGENCE

14. Copyright (C) 2017 451 Research LLC
Batch processing as a barrier to intelligence
SYSTEMS OF ANALYSISSYSTEMS OF RECORD
SYSTEMS OF ENGAGEMENT
DATA
ANALYSTS DATA
SCIENTISTS
DECISION
MAKERS
BUSINESS
USERS
SYSTEMS OF INTELLIGENCE

15. 15
Problems with what’s going on here
What’s slowing the conversion of data to insight?
1. Monolithic ETL systems
2. “Waterfall” changes
3. No shared responsibility
4. Singular unit of scale
5. Testing....???

16. The case for Microservices
for Continuous Data
Integration
1

17. Copyright (C) 2017 451 Research LLC
The Case for Continuous Data Integration
SYSTEMS OF ANALYSISSYSTEMS OF RECORD
SYSTEMS OF ENGAGEMENT
DATA
ANALYSTS DATA
SCIENTISTS
DECISION
MAKERS
BUSINESS
USERS
SYSTEMS OF INTELLIGENCE

18. Copyright (C) 2017 451 Research LLC
Continuous Data Integration
The key concepts of a
continuous integration and
delivery process can be
applied to the process of
developing, deploying and
managing data integration
pipelines, resulting in
pipelines that are responsive
to changing business and
data processing
requirements.

19. for Data Integration
Before Cloud
● Servers were expensive.
● Software was hard to acquire
and slow to deploy.
● Nightly batches
● Integration in large rollouts and
required specialized skills.
Today
● Lower compute cost
● Software is core to business value,
developed in-house.
● Higher data volumes
● Scale-out architectures
● Streaming data sources
Why Microservices
Integration flows can leverage this by operating as microservices
that scale independently, and are reusable, and interoperate over
message queues.

20. Spring Cloud Data Flow
Spring Cloud Data Flow is a Microservices
toolkit for building data integration and real-
time data processing pipelines.
Pipelines consist of Spring Boot apps,
using Spring Cloud Stream for events
or Spring Cloud Task for batch processes.
The Data Flow server provides interfaces
to compose and deploy pipelines onto
a runtime like Kubernetes or Cloud Foundry.

21. Day 2: People and
Platforms
2

22. Copyright (C) 2017 451 Research LLC
What Exactly Needs to be Transformed Digitally?
Process
Transforma-on
Informa-on
Transforma-on
Pla0orm
Transforma-on
! More than a technical shi., but a cultural one
! Focus on collabora3on—employees but also
customers, partners, suppliers
! Agile methods for so.ware development
! Gathering data and lots of it in various means
and methods
! Mul3ple communica3on points on mul3ple
devices
! Leveraging data with advanced analy3cs
! IT moving from cost center to so.ware enabler
! Organiza3ons needs systems of engagement—tools
and systems for omnichannel interac3on
! Integra3on with legacy systems of record

23. 23
Methodologies Deployment
Sparingly at
designated times
Ready for prod at
any time
Architecture Technologies Operations
App Server on Machine
Containers,
Public / Private /
Hybrid Cloud
Monolithic Systems
Microservices /
Composite app
Linear / Sequential
Agile
DevOps
CI / CD Pipelines
Many tools, ad hoc
automation
Manage services,
not servers
Software Engineering Practices are Evolving

24. So, Now What?
2

25. Align your
Data engineering
and development
teams
Identify
data sources,
processes, and
destinations
Focus on
system of
engagement in
need of
intelligence

26. 26
Thank You!