This document discusses Spring Cloud Data Flow, which provides tools for creating complex data pipelines for streaming and batch processes. It describes how Spring Cloud Data Flow orchestrates Spring Cloud Stream for event streaming and Spring Cloud Task for short-lived microservices. The document also references challenges around integration, event streaming, and orchestrating short-lived tasks that Spring Cloud Data Flow addresses.
22. Test Surface Area
• Unit Tests: 60+ Pre-built and Unit Tested Sources, Sinks and
Processors
• Integration Tests: Applications have a defined input/output
• Component Tests: Application is the Component boundary
• Contract Tests: External services are clearly visible and defined
• End to End Tests: This is where testing is needed.
35. framework for building highly
scalable event-driven
microservices connected with
shared messaging systems
Spring Cloud Stream
36. Spring Cloud Stream
Rabbit MQ Apache Kafka Google PubSub
Amazon Kinesis Azure Event Hubs Solace
Pluggable Binder
Implementations
37. Spring Cloud Stream
Pluggable Binder
Implementations
Imperative + Functional
Programming Model
public class TransferServiceImpl implements TransferService {
public TransferServiceImpl(AccountRepository ar) {
this.accountRepository = ar;
}
38. Spring Cloud Stream
@EnableBinding(Processor.class)
public class Application {
@StreamListener("foo")
@SendTo("bar")
public String replaceStringMsgHandler(String payload) {
return StringUtils.replace(payload, "foo", "bar");
}
}
B
I
N
D
I
N
G
E
V
E
N
T
S
foo
channel
B
I
N
D
I
N
G
channel
bar
C
O
N
S
U
M
E
R
S
Message Channel Abstraction
Programming Model
39. Spring Cloud Stream
@EnableBinding(Processor.class)
public class Application {
@StreamListener("foo")
@SendTo("bar")
public Flux<Average> sensorAverage(Flux<Sensor> data) {
return . .;
}
}
E
V
E
N
T
S
B
I
N
D
I
N
G
foo
channel
bar
B
I
N
D
I
N
G
channel
C
O
N
S
U
M
E
R
S
Native Reactor Flux’s
Programming Model
40. Spring Cloud Stream
Programming Model
@EnableBinding(Processor.class)
public class Application {
@StreamListener("foo")
@SendTo("bar")
public KStream<Object, Foo> handler(KStream<Object,
Event> input){
return . .;
}
}
Native Kafka Streams
E
V
E
N
T
S
B
I
N
D
I
N
G
foo
channel
bar
B
I
N
D
I
N
G channel
C
O
N
S
U
M
E
R
S
41. Spring Cloud Stream
Programming Model
@EnableBinding(Processor.class)
public class Application {
@Bean
public Function<String, String> toUpperCase() {
return s -> s.toUpperCase();
}
}
E
V
E
N
T
S
B
I
N
D
I
N
G
foo
channel
bar
B
I
N
D
I
N
G channel
C
O
N
S
U
M
E
R
S
Plain Old Java Functions
42. Spring Cloud Stream
Pluggable Binder
Implementations
Stream
Partitions
Consumer
Groups
Message
Headers
Testing
Framework
Content-type
Negotiation
Imperative + Functional
Programming Model
public class TransferServiceImpl implements TransferService {
public TransferServiceImpl(AccountRepository ar) {
this.accountRepository = ar;
}
45. @EnableTask
public class TimestampTask {
@Bean
public TimestampTask timeStampTask() {
return new TimestampTask();
}
public static class TimestampTask implements CommandLineRunner {
@Override
public void run(String... strings) throws Exception {
DateFormat dateFormat = . .
logger.info(dateFormat.format(new Date()));
}
}
}
Database
R
E
P
O
S
I
T
O
R
Y
An arbitrary business-logic as short-lived Application
@EnableTask
Spring Cloud Task
46. @EnableTask
@EnableBatchProcessing
public class BatchJobApplication {
@Bean
public Step extractStep() { // extract business logic }
@Bean
public Step transformStep() { // transformation logic }
@Bean
public Step loadStep() { // persistence logic }
@Bean
public Job etlJob() {
return this.jobBuilderFactory.get("etlJob")
.start(extractStep())
.next(transformStep())
.next(loadStep())
.build();
}
}
Database
R
E
P
O
S
I
T
O
R
Y
Spring Batch Job as short-lived Application
Spring Cloud Task
47. Spring Cloud Task
Lifecycle Management Transactions
Bookkeeping for Restarts/Replay Historical Representation
Remote
Partitions
51. Spring Cloud Stream -> Task
SFTP Source TaskLauncher ETL Job/Task
Database
Orchestrated by Spring Cloud Data Flow
SFTP Server
poll for
new files
publish each file launch task for each file
persist
parsed data
52. Spring Cloud Task -> Stream
SFTP Source TaskLauncher Batch Job/Task
Database
SFTP Server
poll for
new files
publish each file launch task for each file
persist
parsed data
:task-eventsTaskLauncher
53. Spring Cloud Data Flow
Integration Problems
Event Streaming
Orchestrating Short Lived Tasks
54.
55.
56. References
• Software developers are now more valuable to companies than money - CNBC
• Testing Strategies in a Microservice Architecture by Martin Fowler and Toby
Clemson
• Presentation Patterns by Neal Ford, Matthew McCullough, and Nate Schutta
• Don't Walk Away from Complexity, Run • Venkat Subramaniam
• Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey
• Dave Syer, Pivotal—Choosing Software Abstractions, SpringOne Platform
2018
57. References (con’t)
• Code for Demos: https://github.com/abhinavrau/scdf-demo
• Spring Cloud Data Flow Microsite Reference Doc
• Spring Cloud Stream Samples | Gitter | StackOverflow
• Spring Cloud Task Samples | Gitter | StackOverflow
• Spring Cloud Skipper Samples | Gitter | StackOverflow