Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Orchestrating Data Microservices with Spring Cloud Data Flow

SpringOne Platform 2017
Mark Pollack, Pivotal

"In this session you will learn how you can create data integration and real-time data processing pipelines using Spring Cloud Data Flow and deploy them to multiple platforms – Cloud Foundry, Kubernetes, and YARN to name a few. Spring Cloud Data Flow’s DSL and Designer User Interface will be demonstrated to show how you can easily assemble data pipelines without writing any code for common use-cases. When the need for custom code arises, you can create new application components using the programming model offered by Spring Cloud Stream and Spring Cloud Task.

The latest features of Spring Cloud Data Flow will be showcased, which include managing the entire lifecycle of data processing pipelines, allowing for a subset of the applications that compose a stream to be updated independently or rolled back to previous versions and also data provenance of data flowing through the pipelines."

Orchestrating Data Microservices with Spring Cloud Data Flow

  1. 1. Orchestrating Data Microservices with Spring Cloud Data Flow By Mark Pollack @markpollack 1
  2. 2. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Speaker Background ● Spring Cloud Skipper ● Spring Cloud Data Flow ● Spring XD ● Spring Shell ● Spring Data ● Spring.NET ● Spring Framework 2
  3. 3. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ What is Spring Cloud Data Flow ● Spring Cloud Data Flow lets you easily create data integration processing pipelines for continuous streams of events or in finite batches. ● Pipelines consist of Spring Boot apps, using Spring Cloud Stream for events or Spring Cloud Task for batch processes. ● The Data Flow server provides interfaces to compose and deploy pipelines onto multiple Cloud Platforms using ○ DSLs ○ UI Designer ○ Stream and Task Administration UI 3
  4. 4. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Microservices are not just for web apps ● Spring Cloud Stream ○ Event driven based Microservices ○ Loose coupling via pub/sub messaging ○ Opinionated Primitives ○ For integration and stream processing use-cases 4
  5. 5. Spring Cloud Stream in a nutshell Application Core Messaging Middleware Binder Inputs Output s Spring Boot Configuration Pluggable Messaging Middleware: RabbitMQ, Kafka, Google PubSub, JMS, Kinesis Flexible programming model: Spring Integration, KStream, Reactor Standardized configuration model
  6. 6. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Microservices are not just for web apps ● Spring Cloud Task ○ Tasks are short lived Boot Microservices ○ System tracks invocations, exit-status ○ Spring Batch Jobs are wrapped as Tasks ○ Useful for ETL between databases and filesystems 6
  7. 7. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data Flow Architecture 7 SCDF Server sour ce Deployer Implementation
  8. 8. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Don’t start from scratch! ● Choose from one of the app starters ● Or create a custom stream or task application from Spring Initializr 8
  9. 9. UI Tour
  10. 10. Let’s do a Twitter Sentiment Analysis!
  11. 11. Ingest Scoring Storag e
  12. 12. 1 User submits a DSL to SCDF 2 Parse the DSL; resolve Spring Boot Apps 3 Push Apps; bind services; stage; deploy 4 Operational metrics and monitoring Inges t Filter Predic t Stor e Data Flow Lifecycle
  13. 13. Java DSL
  14. 14. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Tasks - File Ingest 14 FTP Server sftpLister | batchJobTransformer --uri=batch-ingest-1.0.0.jar > :filesAvailable FTPPollStream BatchIngestTask DB LaunchStream :filesAvailable > task-launcher-local
  15. 15. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Some Feedback ● Changing deployment properties or app versions means deploying a completely new stream ● Rely on runtime-platform’s blue-green deployment support for rolling- upgrades 15
  16. 16. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Stream Updates 16
  17. 17. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Stream Updates 17
  18. 18. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Introducing Spring Cloud Skipper Skipper is a tool that allows you to ● Search for Spring Boot applications ● Manage their lifecycle on multiple Cloud Platforms Use standalone or within CI pipeline Latest release 1.0 M3 http://cloud.spring.io/spring-cloud-skipper/ 18
  19. 19. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data Flow Architecture with Skipper 19 SCDF Server Deployer Implementations Skipper Server sour ce Generate Skipper Package
  20. 20. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Easy Upgrades and Rollbacks 20 app register --name http --type source --uri maven://org.springframework.cloud.stream.app:http-source-rabbit:1.2.0.RELEASE app register --name log --type sink --uri maven://org.springframework.cloud.stream.app:log-sink-rabbit:1.1.0.RELEASE
  21. 21. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Easy Upgrades and Rollbacks 21 stream create --name httptest --definition "http | log" stream skipper deploy --name httptest stream skipper update --name httptest --properties version.log=1.2.0.RELEASE stream skipper rollback --name httptest make improvement to log sink, release version 1.2.0
  22. 22. Data Flow With Skipper
  23. 23. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Functions ● Write a java.util.Function for Stream Sources, Processors, and Sinks 23 public class CharCounter implements Function<String, Integer> { @Override public Integer apply(String word) { return word.length(); } }
  24. 24. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Functions 24 http --server.port=9001 | function-runner --function.className=com.example.functions.CharCounter --function.location=file:///<PATH/TO/FUNCTION>/myfunction.jar | log app register --name function-runner --type processor --uri http://repo.spring.io/.../function-app-rabbit-1.0.0.jar
  25. 25. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Roadmap: https://buzz.cfapps.io 25
  26. 26. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Roadmap: Improved Samples https://docs.spring.io/spring-cloud- dataflow/docs/1.3.0.M3/reference/htmlsingle/#dataflow-samples 26
  27. 27. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ https://cloud.spring.io/spring-cloud-dataflow 27
  28. 28. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ http://cloud.spring.io/spring-cloud-skipper/ ● https://gitter.im/spring-cloud/spring-cloud-skipper 28
  29. 29. Thanks! 29 #springone@s1p
  30. 30. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ STOP! Download Fonts Now PLEASE DOWNLOAD AND INSTALL PROXIMA NOVA FONTS BEFORE CREATING YOUR PRESENTATION. You can download the fonts here… https://brandfolder.com/pivotal Password: keepitsimple Fonts included in the ZIP file: Proxima Nova (headline and body text) http://www.fontspring.com/support/installing/how-do-i-install-fonts-on-my-mac http://www.fontspring.com/support/installing/how-do-i-install-fonts-on-my-windows- pc 30
  31. 31. Java DSL
  32. 32. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation. 32 If you are a Pivotal employee and your talk contains forward facing information, please include this slide at the beginning or conclusion of the presentation. Remove this text box before presenting.
  33. 33. Slide Templates
  34. 34. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Body Slide - Dark Background All body text is Proxima Nova Regular • Subhead (18pt) • Level Two (18pt) • Level Three (18pt) • Level Four (18pt) Use the “Decrease/Increase Indent” tools to change bullet levels • Click on the Home ribbon, Paragraph tab Line spacing is set in master slides 34
  35. 35. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Two Columns – Dark Background Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque 35
  36. 36. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Body Slide - Light Background 36 All body text is Proxima Nova Regular • Subhead (18pt) • Level Two (18pt) • Level Three (18pt) • Level Four (18pt) Use the “Decrease/Increase Indent” tools to change bullet levels • Click on the Home ribbon, Paragraph tab Line spacing is set in master slides
  37. 37. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Two Columns – Light Background Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque 37
  38. 38. Design Assets
  39. 39. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Brand Colors SpringOne Platform 2017 Theme Colors 39 Spring Brand Color Pivotal Brand Color
  40. 40. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Code Slide 40 // This is Andale Mono: 14pt or higher please public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; } … }
  41. 41. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Pivotal Logos on Dark Background 41 Looking for more Pivotal logos, PCF services icons, or OSS logos? Visit: brandfolder.com/pivotal-assets
  42. 42. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Pivotal Logos on Light Background 42 Looking for more Pivotal logos, PCF services icons, or OSS logos? Visit: brandfolder.com/pivotal-assets
  43. 43. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Event Logos 43
  44. 44. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Logo and Project Icons 44 Spring Framewor k Spring Securit y Sprin g Data Sprin g Batch Spring Integratio n Project Reacto r Sprin g AMQ P Spring Hateoa s Spring Mobil e Spring Androi d Sprin g Social Spring Web Services Spring Web Flow Sprin g XD Sprin g Boot Sprin g LDAP Spring Tool Suite Spring Cloud Data Flow Spring Kafka Spring Cloud
  45. 45. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Sample Table 45 2007 2008 2009 2010 Category 1 2.4 6.4 4.1 6.6 Category 2 8.2 4.5 3.2 3.8 Category 3 4.6 3.2 1.9 9.6 Category 4 6.7 3.3 3.4 2.2 Category 5 4.3 5.6 7.1 3.4
  46. 46. Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Sample Table 46 2007 2008 2009 2010 Category 1 2.4 6.4 4.1 6.6 Category 2 8.2 4.5 3.2 3.8 Category 3 4.6 3.2 1.9 9.6 Category 4 6.7 3.3 3.4 2.2 Category 5 4.3 5.6 7.1 3.4
  47. 47. Learn More. Stay Connected. <Your CTA> <Related Session> 47 #springone@s1p

×