Delivering Cloud Native Batch Solutions with Dodd Pfeffer at SpringOne Tour 2019

1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.
Dodd Pfeffer
Pivotal Platform Architect
Cloud Native Batch

3. Use Cases
3

4. Modernize monolithic
ETL workloads
SQL scripts, stored
procedures, and in-house
bash scripts to cloud-
native architecture.
Small and incremental
releases.
Continuous delivery is the
focus.
4

5. Scheduled batch-jobs
Whether it is for
predictive model training,
massive file movement, or
the classic data migration
batch-jobs, they are
typically schedule driven.
5

6. Dragonstone Finance
● Good DDD design approaches
● Internal ratings service
○ Influenced by a series of analyst
evaluations provided via file integration
from various sources
● Internal auditing function
○ Accepts trade data via file integration from
various sources
○ Provides feedback to traders and trading
system on healthiness of trades

7. What would a
cloud native
batch solution
look like
● Batch apps run on ephemeral containers whose capacity is only
consumed during batch execution
● Polling for new files delivered by data microservice pipelines via
ingest streams
● Scheduler provided by the platform
● Batch app properties externally managed through config server
● Batch app external service dependency facilitated through
service registry
● Cloud platform dynamically binds persistent volume-service to
container during batch execution
● Database connection information dynamically injected into
batch app environment
● Databases provisioned and managed by via cloud platform
● Batch app logs aggregated and stored on the platform, optional
streamed to external log source
● Access to manage and monitor the batch applications
consistent with other microservices on the platform

8. Dragonstone Finance
● Spring Batch
● Spring Cloud Task
● Spring Cloud Data Flow
● Spring Cloud Config Server
● Spring Cloud Service Registry
● Pivotal Application Services
● Volume Services for PCF
● MySql for PCF

9. File Ingest Reference Architecture
https://content.pivotal.io/blog/need-24x7-etl-then-move-to-cloud-native-file-ingest-with-spring-cloud-data-flow

10. Spring Batch
● Lightweight framework
● Familiar spring interfaces
● Transaction management
● Chunk based processing
● Declarative I/O
● Start/Stop/Restart
● Retry/Skip
https://docs.spring.io/spring-batch/4.1.x/reference/html/index-single.html

11. Spring Cloud Stream
● Stream is composed of long lived Boot Microservice apps
● Event driven based Microservices
● Loose coupling via pub/sub messaging
● Opinionated Middleware Primitives
● For integration and stream processing use-cases
● Each microservice app can be packaged as an uber-jar or as a Docker container
● Out-of-the-box apps: https://cloud.spring.io/spring-cloud-stream-app-starters/

12. Spring Cloud Task
● Tasks are short lived Boot Microservice apps
● System tracks invocations, exit-status
● Spring Batch Jobs are wrapped as Tasks
● Useful for ETL between databases and filesystems
● Each microservice app can be packaged as an uber-jar or as a Docker container
● Out-of-the-box apps: https://cloud.spring.io/spring-cloud-task-app-starters/

13. Spring Cloud Stream
● Spring Boot Application that stays on
listening for a data source.
● Provides abstraction to a message broker
implementation.
Stream Versus Task
Spring Cloud Task
● Spring Boot application that terminates its
process when it has completed its task.
● Implements a CommandLineRunner bean
● May publish messages using Spring Cloud
Stream.
● Maintains its own task repository to track
task analytics

14. What is Spring Cloud Data Flow?
Spring Cloud Data Flow is a Microservices toolkit
for building data integration and real-time data
processing pipelines.
Pipelines consist of Spring Boot apps, using
Spring Cloud Stream for events or Spring Cloud
Task for batch processes.
The Data Flow server provides interfaces to
compose and deploy pipelines onto a modern
platform like Cloud Foundry.

15. Cover w/ Image
Spring Cloud Services for Pivotal Cloud Foundry
■ Pivotal has bundled a number of these
OSS projects into Spring Cloud Services
for PCF
Service Registry
Config Server
Circuit Breaker Dashboard

16. Increase speed and deploy code to
production thousands of times per
month. Use PAS to run Java, .NET,
and Node apps.
Pivotal Application Service (PAS): A Runtime for Apps
Best runtime for Spring and Spring Boot — Spring’s
microservice patterns—and Spring Boot’s executable jars—are
ready-made for PAS.
Turnkey microservices operations and security — Spring
Cloud Services brings microservices best practices to PAS. It
includes Config Server, Service Registry, and Circuit Breaker
Dashboard.
A native Windows and .NET experience — Use PAS to run new
apps built with .NET Core. Run your legacy .NET Framework
apps on PAS too, using the .NET Hosted Web Core buildpack.
Push applications to containers running on Windows Server
2016.
Built for apps — PAS has everything to need to run apps.
Buildpacks manage runtime dependencies; metrics, logging,
and scaling are done for you. Multitenancy, and blue/green
deployment patterns are built-in. Extend apps with a rich service
catalog.
Container-ready — PAS supports the OCI format for Docker
images. Run platform-built and developer-built containers.

17. Let’s take a look

18. References
● 24x7 ETL / File Ingest Reference Architecture: https://content.pivotal.io/blog/need-24x7-etl-then-
move-to-cloud-native-file-ingest-with-spring-cloud-data-flow
● Spring Batch Docs:
https://docs.spring.io/spring-batch/4.1.x/reference/html/index-single.html
● Spring Cloud Data Flow Docs: https://docs.spring.io/spring-cloud-
dataflow/docs/current/reference/htmlsingle/
● Volume Services: https://docs.pivotal.io/pivotalcf/2-5/devguide/services/using-vol-
services.html#create-nfs
● Demo Source Repo:
https://github.com/doddatpivotal/dragonstone-finance

19. Register Today Early Bird ends 7/8
Use
Discount Code
S1P_Save200
October 7–10, 2019
Austin Convention Center
and Save!

20. Transforming How The World Builds Software
© Copyright 2017 Pivotal Software, Inc. All rights Reserved.Revised 1/25/19