Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow and Micrometer by Christian Tzolov at SpringOne Tour 2019

1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.
Christian Tzolov (@christzolov)
March 2019
Real-time Analysis of Data
Processing Pipelines
Spring Cloud Data Flow & Micrometer

2. About @christzolov
Pivotal engineer, Spring Cloud Data Flow team
Apache Committer/PMC member
Distributed, Data-intensive Systems and Toolkits

3. Agenda ■ Distributed, Data-Intensive Systems
■ Spring Cloud Data Flow toolkit
■ Operational metrics and monitoring
■ Micrometer, Time Series and Dimensions
■ Architectural Patterns and Practices
■ Q+A

4. “We call an application data-intensive if data is its
primary challenge—the quantity of data, the
complexity of data, or the speed at which it is
changing.”

5. What is Spring Cloud Data Flow
A toolkit for building
data integration, real-
time streaming, and
batch data
processing pipelines.

6. What is Spring Cloud Data Flow
A toolkit for building
data integration, real-
time streaming, and
batch data
processing pipelines.
Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60 out-of-the-
box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Continuous delivery for streaming data pipelines
using Spring Cloud Skipper.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.

7. What is Spring Cloud Data Flow
Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60 out-of-the-
box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Continuous delivery for streaming data pipelines
using Spring Cloud Skipper.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.
A toolkit for building
data integration, real-
time streaming, and
batch data
processing pipelines.

8. What is Spring Cloud Data Flow
Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60 out-of-the-
box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Continuous delivery for streaming data pipelines
using Spring Cloud Skipper.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.
A toolkit for building
data integration, real-
time streaming, and
batch data
processing pipelines.

9. What is Spring Cloud Data Flow
Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60 out-of-the-
box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Continuous delivery for streaming data pipelines
using Spring Cloud Skipper.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.
A toolkit for building
data integration, real-
time streaming, and
batch data
processing pipelines.

10. What is Spring Cloud Data Flow
Data pipelines consist of Spring Boot apps, using
Spring Cloud Stream for event-streaming or
Spring Cloud Task for batch processes.
Ready for Data Integration with >60 out-of-the-
box streaming and batch Apps.
DSL, GUI, and REST-APIs to build and
orchestrate data pipelines onto platforms like
Kubernetes and Cloud Foundry.
Continuous delivery for streaming data pipelines
using Spring Cloud Skipper.
Cron-job scheduler for batch data pipelines using
Spring Cloud Scheduler.
A toolkit for building
data integration, real-
time streaming, and
batch data
processing pipelines.

11. Runtime and Message Broker Abstraction
Kubernetes
Cloud Foundry
Local / Dev
Rabbit MQ Apache Kafka Google PubSub
Amazon Kinesis Solace
Opportunities: Same code; Same tests; Works with a variety of Message Brokers

12. Common Denominator = Spring Boot => Consolidate On
Development Practices
Test Infrastructure
CI/CD Tooling and Automation
Operational Metrics and Monitoring

13. Data Processing Pipeline

14. Data Flow
Lifecycle
• Data Flow accepts the data
pipeline definition and
delegates to Skipper for
lifecycle managements.
• Operational metrics and
Monitoring.

15. Legacy
Spring Cloud Metrics Collector

16. What we want
Micrometer
Time Series (Influx, Prometheus)
Grafana

17. Time Series &
Dimensional Data
Model
● A time series is a series of data points ordered in
time order.
● timestamped values belonging to the same
metric and the same set of labeled dimensions.
● Every time series is uniquely identified by
its metric name and a set of key-value pairs,
known as labels, tags, dimensions.

18. Data Flow 2.0
Monitoring
Architecture
Support for Prometheus and
InfluxDB

19. Data Pipeline Monitoring

20. Supported
Metrics
● Spring Boot 2.0 - JVM, CPU, File descriptor…
● Spring Integration - Channel, Source, Handler
● Data Flow App Starters – Tags: stream name,
app name, type, CF
process_cpu_usage {
application_guid="20036",
application_name="log",
application_type="sink",
instance_index="0",
stream_name="s3",
instance="10.40.1.170:20036",
job="scdf",
}
spring_integration_send_seconds_count {
application_guid="20036",
application_name="log",
application_type="sink",
stream_name="s3",
instance_index="0",
name="input",
result="success",
type="channel"
exception="none",
instance="10.40.1.170:20036",
job="scdf",
}

21. Pull-based TSDB
(Prometheus)
Implications:
• Service Discovery
• Security
• Short Living Tasks
(PushGateway)

22. Data Flow in
Kubernetes
Prometheus

23. Data Flow in PCF
InfluxDB
PromRegator

24. SCDF Monitoring with hosted Grafana and InfluxDB

25. Data Flow
Analytics with
Micrometer
Counter Processor & Sink

26. Twitter Analytics

27. Next Steps
● Spring Cloud Stream Task support
● Monitor Data Flow and Skipper
● PromRegator and PCF improvements
● ?!

28. References
● Spring Cloud Data Flow – Stream Monitoring: http://docs.spring.io/spring-cloud-
dataflow/docs/2.1.0.BUILD-SNAPSHOT/reference/htmlsingle/#streams-monitoring
● Twitter Analytics Sample: https://docs.spring.io/spring-cloud-dataflow-
samples/docs/current/reference/htmlsingle/#spring-cloud-data-flow-samples-twitter-analytics-
overview
● Micrometer: https://micrometer.io
● Spring Boot Micrometer : https://docs.spring.io/spring-
boot/docs/2.1.3.RELEASE/reference/htmlsingle/#production-ready-metrics
● Spring Integration – Micrometer: https://docs.spring.io/spring-
integration/docs/5.1.2.RELEASE/reference/html/system-management-chapter.html#micrometer-
integration
● Grafana: https://grafana.com
● Prometheus: https://prometheus.io

29. Q&A
SCDF STREAM MONITORING