This document summarizes a presentation about evolving legacy systems to a modern, event-driven architecture using Apache Kafka and change data capture. It discusses moving from a tangled "spaghetti architecture" to a message-driven system using microservices and APIs. It also covers offloading legacy data and processes from mainframes and other systems by reading change logs with Attunity Replicate to minimize disruption. The talk presents an evolutionary approach and maturity model for offloading from the data level to fully migrating business processes and applications.

1. Milano Apache Kafka Meetup

2. Milano Apache Kafka® Meetup / quantyca.it
www.quantyca.it
www.blindata.io

3. How to integrate legacy systems
within a modern polyglot and event
driven architecture using an
evolutionary strategy
Milano Apache Kafka® Meetup / quantyca.it
Milano, 4.07.2018

4. Agenda
Milano Apache Kafka® Meetup / quantyca.it
4
First steps in the
evolutionary journey
1
Why we need to evolve
3
The destination:
message driven
architecture
5
Legacy offloading
6
Attunity replicate
and the CDC maturity
model
2
The starting point:
spaghetti architecture

5. We really need to evolve?
Milano Apache Kafka® Meetup / quantyca.it
Martec’s Law
“The world is changing
very fast. Big will not beat
small anymore. It will be
the fast beating the slow.”
Rupert Murdoch

6. Evolve or die
Milano Apache Kafka® Meetup / quantyca.it

7. Agenda
Milano Apache Kafka® Meetup / quantyca.it
4
First steps in the
evolutionary journey
3
The destination:
message driven
architecture
5
Legacy offloading
6
Attunity replicate
and the CDC maturity
model
1
Why we need to evolve
2
The starting point:
spaghetti architecture

8. You are here: spaghetti architecture
Milano Apache Kafka® Meetup / quantyca.it
@see
Big Ball of Mud

9. Driving force: Polyglotism
9
Polyglot Programming
Applications should be written in a mix of
languages to take advantage of the fact that
different languages are suitable for tackling
different problems. Complex applications
combine different types of problems, so
picking the right language for the job may be
more productive than trying to fit all aspects
into a single language
2006
Polyglot Persistence
A complex enterprise application uses different
kinds of data, and already usually integrates
information from different sources. Increasingly
we'll see such applications manage their own
data using different technologies depending on
how the data is used. This trend will be
complementary to the trend of breaking up
application code into separate components
integrating through web services
2011
Milano Apache Kafka® Meetup / quantyca.it
Polyglotism without a clear IT strategy has increased the architectural technical debt
@see
Polyglot Persistence by Martin Fowler

10. Agenda
Milano Apache Kafka® Meetup / quantyca.it
4
First steps in the
evolutionary journey
3
The destination:
message driven
architecture
1
Why we need to evolve
5
Legacy offloading
6
Attunity replicate
and the CDC maturity
model
2
The starting point:
spaghetti architecture

11. Message driven systems to the rescue
Reactive Systems rely on asynchronous message-
passing to establish a boundary between components
that ensures loose coupling, isolation and location
transparency.
This boundary also provides the means to delegate
failures as messages. Employing explicit message-passing
enables load management, elasticity, and flow control by
shaping and monitoring the message queues in the
system and applying back-pressure when necessary.
Location transparent messaging as a means of
communication makes it possible for the management of
failure to work with the same constructs and semantics
across a cluster or within a single host. Non-blocking
communication allows recipients to only consume
resources while active, leading to less system overhead.
The Reactive Manifesto
Milano Apache Kafka® Meetup / quantyca.it
@see
https://www.reactivemanifesto.org/

12. Message driven architectures
Event-driven architectures (EDA) usually integrate several disparate systems together using message queues.
There are two common implementations of this type of architecture:
Milano Apache Kafka® Meetup / quantyca.it
broker pattern mediator pattern
@see
Variations in event-driven architecture: Find out if mediator or broker topology is right for you.

13. Traditional SOA (the evil) is an example of mediator pattern
In building communication structures between different processes, SOA approach put significant smarts into the
communication mechanism itself (smart pipe, dumb endpoints)
Milano Apache Kafka® Meetup / quantyca.it
=
Spaghetti architecture Spaghetti in a box architecture
@see
Does My Bus Look Big in This?

14. The real problem with traditional SOA
The problem with traditional SOA is not anyway the use of the mediator pattern but the degree to which developers
have deviated from bounded context. This is the key element that largely determines the quantum size and how much
damaging coupling appears in the overall architecture.
Enterprise services Share nothing domain services
>
coupling
continuum
Milano Apache Kafka® Meetup / quantyca.it
Traditional SOA Microservices
@see
Microservices as an Evolutionary Architecture

15. Software architectures aren’t created in a vacuum
If there are clear benefits, then why haven’t developers embraced this style before?
A decade ago, automatic provisioning of machines wasn’t possible. Operating systems were
commercial and licensed, with little support for automation. Real-world constraints like
budgets impact architectures, which is one of the reasons developers build more and more
elaborate shared resources architectures, segregated at the technical layers. If operations is
expensive and cumbersome, architects build around it, as they did in ESB-SOAs.
Milano Apache Kafka® Meetup / quantyca.it

16. Benefit and tradeoffs
Developers understand the benefits of everything and the
tradeoffs of nothing!
Rich Hickey, creator of Clojure
Milano Apache Kafka® Meetup / quantyca.it
Architects must understand both benefits and
tradeoffs of microservices and build engineering
practices accordingly.
Despite their popularity microservices are not the
right solution for any kind of problems

17. Alternative to microservices
Relaxing share nothing principle
A more commonly used architectural style for migration is a service-based architecture, which is similar to but could differ
from microservices in three important ways:
1. service granularity,
2. database scope,
3. integration middleware.
Service-based architectures are still domain-centric but address some challenges developers face when restructuring
existing applications toward more evolutionary architectures.
Microservices and service based architectures can cohesist. It’s a responsibility of the system architects to decide analyzing
the different use cases when to go with one solution and when to go with the other.
Milano Apache Kafka® Meetup / quantyca.it
@see
Service-Based Architecture as an Alternative to Microservice Architecture

18. Example of an hybrid service architecture
Milano Apache Kafka® Meetup / quantyca.it
Service based architecture and the service bus are used
when
1. control is more important than scalability
2. there are transaction that span bounded contexts
3. there are not enough skilled developers to implement
all with microservices
4. there is a monolith that requires an anti-corruption
layer
5. there is a legacy database that cannot be splitted
between microservices
Microservices architecture and the data bus are used in
all other cases
An API gateway hides to the callers if it are calling a service
through the service bus or the data bus
@see
Quantyca’s Reference Implementation

19. Agenda
Milano Apache Kafka® Meetup / quantyca.it
4
The starting point:
spaghetti architecture
3
The destination:
message driven
architecture
5
Legacy offloading
6
Attunity replicate
and the CDC maturity
model
1
Why we need to evolve
2
First steps in the
evolutionary journey

20. How to evolve from spaghetti architecture?
Milano Apache Kafka® Meetup / quantyca.it

21. Step 1: Working at the edge
Milano Apache Kafka® Meetup / quantyca.it
@see
Microservice Prerequisites

22. Step 2: Refactoring monoliths
Milano Apache Kafka® Meetup / quantyca.it
@see
How to break a Monolith into Microservices

23. Refactoring monoliths
Milano Apache Kafka® Meetup / quantyca.it

24. Agenda
Milano Apache Kafka® Meetup / quantyca.it
4
First steps in the
evolutionary journey
5
Legacy offloading
1
Why we need to evolve
3
The destination:
message driven
architecture
6
Attunity replicate
and the CDC maturity
model
2
The starting point:
spaghetti architecture

25. Vendor king antipattern
The viral lock-in
Many organizations become overambitious with some legacy software (i.e. ERP, Mainframe, ecc ..), leading to the vendor
king antipattern, an architecture built entirely around a vendor product that pathologically couples the
organization to a tool.
By placing an external tool or framework at the heart of the architecture, developers severely restrict their ability to
evolve in two key ways,
■ TECHNICAL PROCESS
From a business process standpoint the tool whole architecture depends by choices the vendor makes in
terms of persistence, supported infrastructure, and a host of other constraints
■ BUSINESS PROCESS
From a business process standpoint, the tool simply can’t support the optimal workflow; this is a side
effect of the Last 10% Trap. Most companies end up knuckling under the framework, modifying their
processes rather than trying to customize the tool. The more companies do that, the less differentiators
exist between companies, which is fine as long as that differentiation isn’t a competitive advantage.
Milano Apache Kafka® Meetup / quantyca.it

26. To big to remove
Milano Apache Kafka® Meetup / quantyca.it
Having developed and optimized Legacy King Systems for
decades, enterprises are looking for ways to capitalize on these
investments with the ability to reuse, repurpose, and integrate
them in support of new business and IT initiatives.

27. Legacy offloading
Milano Apache Kafka® Meetup / quantyca.it
DB2/MF
IMS
VSAM
SAP
Legacy King
Data
Warehouse
Straem
Analytics
Hybrid Cloud
Microservices
C-Level /
Menagement
Customer
LoB
Analyst
WHERE DATA NEED TO BE
KAFKA
OFFLOADING

28. Offloading via batch loading
● Queries based on timestamp orchestrated with custom ETL Jobs
● Consume significant processing power on the source system
● Usually executed during “batch windows” (not responsive)
● Unable to manage easily changes in the schema structure
● Custom error handling and failover management
Milano Apache Kafka® Meetup / quantyca.it
@see
Streaming change data capture

29. Offloading via CDC
Milano Apache Kafka® Meetup / quantyca.it
● Read incremental changes directly from the transaction log
● Minimize disruption to production workloads
● Fast updates (near real-time)
● Manage easily changes in the schema structure
● Mature market
@see
Streaming change data capture

30. Agenda
Milano Apache Kafka® Meetup / quantyca.it
4
First steps in the
evolutionary journey
3
The destination:
message driven
architecture
5
Legacy offloading
2
The starting point:
spaghetti architecture
1
Why we need to evolve
6
Attunity replicate
and the CDC maturity
model

31. Attunity Replicate
Milano Apache Kafka® Meetup / quantyca.it
@see
Attunity Replicate Home Page

32. Attunity: workflow
Milano Apache Kafka® Meetup / quantyca.it
STEP 1: Create connections STEP 2: Create task STEP 3: Define task

33. Attunity: workflow
Milano Apache Kafka® Meetup / quantyca.it
STEP 4: Run task

34. Attunity: Source and targets
Milano Apache Kafka® Meetup / quantyca.it

35. Attunity: Kafka target
Milano Apache Kafka® Meetup / quantyca.it

36. Attunity: business cases
Milano Apache Kafka® Meetup / quantyca.it

37. Offloading maturity model
Milano Apache Kafka® Meetup / quantyca.it
Data offloading Process offloading Business offloading

38. Thanks!
Questions?
Milano Apache Kafka® Meetup / quantyca.it
Milano, 4.07.2018

39. Corso Milano, 45 / 20900 Monza (MB)
T. +39 039 9000 210 / F. +39 039 9000 211 / @ info@quantyca.it
www.quantyca.it