Submit Search
Upload
Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
•
20 likes
•
22,424 views
Toshiaki Maki
Follow
Spring Cloud Data Flow #jsug #springday
Read less
Read more
Technology
Report
Share
Report
Share
1 of 117
Download now
Download to read offline
Recommended
Yahoo! JAPANのコンテンツプラットフォームを支えるSpring Cloud Streamによるマイクロサービスアーキテクチャ #jsug #sf_52
Yahoo! JAPANのコンテンツプラットフォームを支えるSpring Cloud Streamによるマイクロサービスアーキテクチャ #jsug #sf_52
Yahoo!デベロッパーネットワーク
Apache Kafkaって本当に大丈夫?~故障検証のオーバービューと興味深い挙動の紹介~
Apache Kafkaって本当に大丈夫?~故障検証のオーバービューと興味深い挙動の紹介~
NTT DATA OSS Professional Services
[Aurora事例祭り]Amazon Aurora を使いこなすためのベストプラクティス
[Aurora事例祭り]Amazon Aurora を使いこなすためのベストプラクティス
Amazon Web Services Japan
DBREから始めるデータベースプラットフォーム
DBREから始めるデータベースプラットフォーム
Insight Technology, Inc.
CSI Driverを開発し自社プライベートクラウドにより適した安全なKubernetes Secrets管理を実現した話
CSI Driverを開発し自社プライベートクラウドにより適した安全なKubernetes Secrets管理を実現した話
Katsuya Yamaguchi
実運用して分かったRabbit MQの良いところ・気をつけること #jjug
実運用して分かったRabbit MQの良いところ・気をつけること #jjug
Yahoo!デベロッパーネットワーク
Infrastructure as Code (IaC) 談義 2022
Infrastructure as Code (IaC) 談義 2022
Amazon Web Services Japan
kubernetes初心者がKnative Lambda Runtime触ってみた(Kubernetes Novice Tokyo #13 発表資料)
kubernetes初心者がKnative Lambda Runtime触ってみた(Kubernetes Novice Tokyo #13 発表資料)
NTT DATA Technology & Innovation
Recommended
Yahoo! JAPANのコンテンツプラットフォームを支えるSpring Cloud Streamによるマイクロサービスアーキテクチャ #jsug #sf_52
Yahoo! JAPANのコンテンツプラットフォームを支えるSpring Cloud Streamによるマイクロサービスアーキテクチャ #jsug #sf_52
Yahoo!デベロッパーネットワーク
Apache Kafkaって本当に大丈夫?~故障検証のオーバービューと興味深い挙動の紹介~
Apache Kafkaって本当に大丈夫?~故障検証のオーバービューと興味深い挙動の紹介~
NTT DATA OSS Professional Services
[Aurora事例祭り]Amazon Aurora を使いこなすためのベストプラクティス
[Aurora事例祭り]Amazon Aurora を使いこなすためのベストプラクティス
Amazon Web Services Japan
DBREから始めるデータベースプラットフォーム
DBREから始めるデータベースプラットフォーム
Insight Technology, Inc.
CSI Driverを開発し自社プライベートクラウドにより適した安全なKubernetes Secrets管理を実現した話
CSI Driverを開発し自社プライベートクラウドにより適した安全なKubernetes Secrets管理を実現した話
Katsuya Yamaguchi
実運用して分かったRabbit MQの良いところ・気をつけること #jjug
実運用して分かったRabbit MQの良いところ・気をつけること #jjug
Yahoo!デベロッパーネットワーク
Infrastructure as Code (IaC) 談義 2022
Infrastructure as Code (IaC) 談義 2022
Amazon Web Services Japan
kubernetes初心者がKnative Lambda Runtime触ってみた(Kubernetes Novice Tokyo #13 発表資料)
kubernetes初心者がKnative Lambda Runtime触ってみた(Kubernetes Novice Tokyo #13 発表資料)
NTT DATA Technology & Innovation
SPAセキュリティ入門~PHP Conference Japan 2021
SPAセキュリティ入門~PHP Conference Japan 2021
Hiroshi Tokumaru
NGINXをBFF (Backend for Frontend)として利用した話
NGINXをBFF (Backend for Frontend)として利用した話
Hitachi, Ltd. OSS Solution Center.
20190828 AWS Black Belt Online Seminar Amazon Aurora with PostgreSQL Compatib...
20190828 AWS Black Belt Online Seminar Amazon Aurora with PostgreSQL Compatib...
Amazon Web Services Japan
ブレソルでテラバイト級データのALTERを短時間で終わらせる
ブレソルでテラバイト級データのALTERを短時間で終わらせる
KLab Inc. / Tech
コンテナとimmutableとわたし。あとセキュリティ。(Kubernetes Novice Tokyo #15 発表資料)
コンテナとimmutableとわたし。あとセキュリティ。(Kubernetes Novice Tokyo #15 発表資料)
NTT DATA Technology & Innovation
20190911 AWS Black Belt Online Seminar AWS Batch
20190911 AWS Black Belt Online Seminar AWS Batch
Amazon Web Services Japan
Oracle Database (CDB) on Docker を動かしてみる
Oracle Database (CDB) on Docker を動かしてみる
オラクルエンジニア通信
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
NTT DATA Technology & Innovation
忙しい人の5分で分かるMesos入門 - Mesos って何だ?
忙しい人の5分で分かるMesos入門 - Mesos って何だ?
Masahito Zembutsu
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
NTT DATA Technology & Innovation
Java EE から Quarkus による開発への移行について
Java EE から Quarkus による開発への移行について
Shigeru Tatsuta
マイクロサービス化設計入門 - AWS Dev Day Tokyo 2017
マイクロサービス化設計入門 - AWS Dev Day Tokyo 2017
Yusuke Suzuki
オープンソースのAPIゲートウェイ Kong ご紹介
オープンソースのAPIゲートウェイ Kong ご紹介
briscola-tokyo
HDFSのスケーラビリティの限界を突破するためのさまざまな取り組み | Hadoop / Spark Conference Japan 2019 #hc...
HDFSのスケーラビリティの限界を突破するためのさまざまな取り組み | Hadoop / Spark Conference Japan 2019 #hc...
Yahoo!デベロッパーネットワーク
pgvectorを使ってChatGPTとPostgreSQLを連携してみよう!(PostgreSQL Conference Japan 2023 発表資料)
pgvectorを使ってChatGPTとPostgreSQLを連携してみよう!(PostgreSQL Conference Japan 2023 発表資料)
NTT DATA Technology & Innovation
KafkaとAWS Kinesisの比較
KafkaとAWS Kinesisの比較
Yoshiyasu SAEKI
これで怖くない!?大規模環境で体験するDB負荷対策~垂直から水平の彼方へ~
これで怖くない!?大規模環境で体験するDB負荷対策~垂直から水平の彼方へ~
hideakikabuto
分散ストレージソフトウェアCeph・アーキテクチャー概要
分散ストレージソフトウェアCeph・アーキテクチャー概要
Etsuji Nakai
CRX: Container Runtime Executive
CRX: Container Runtime Executive
imurata8203
分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)
NTT Communications Technology Development
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Toshiaki Maki
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Toshiaki Maki
More Related Content
What's hot
SPAセキュリティ入門~PHP Conference Japan 2021
SPAセキュリティ入門~PHP Conference Japan 2021
Hiroshi Tokumaru
NGINXをBFF (Backend for Frontend)として利用した話
NGINXをBFF (Backend for Frontend)として利用した話
Hitachi, Ltd. OSS Solution Center.
20190828 AWS Black Belt Online Seminar Amazon Aurora with PostgreSQL Compatib...
20190828 AWS Black Belt Online Seminar Amazon Aurora with PostgreSQL Compatib...
Amazon Web Services Japan
ブレソルでテラバイト級データのALTERを短時間で終わらせる
ブレソルでテラバイト級データのALTERを短時間で終わらせる
KLab Inc. / Tech
コンテナとimmutableとわたし。あとセキュリティ。(Kubernetes Novice Tokyo #15 発表資料)
コンテナとimmutableとわたし。あとセキュリティ。(Kubernetes Novice Tokyo #15 発表資料)
NTT DATA Technology & Innovation
20190911 AWS Black Belt Online Seminar AWS Batch
20190911 AWS Black Belt Online Seminar AWS Batch
Amazon Web Services Japan
Oracle Database (CDB) on Docker を動かしてみる
Oracle Database (CDB) on Docker を動かしてみる
オラクルエンジニア通信
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
NTT DATA Technology & Innovation
忙しい人の5分で分かるMesos入門 - Mesos って何だ?
忙しい人の5分で分かるMesos入門 - Mesos って何だ?
Masahito Zembutsu
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
NTT DATA Technology & Innovation
Java EE から Quarkus による開発への移行について
Java EE から Quarkus による開発への移行について
Shigeru Tatsuta
マイクロサービス化設計入門 - AWS Dev Day Tokyo 2017
マイクロサービス化設計入門 - AWS Dev Day Tokyo 2017
Yusuke Suzuki
オープンソースのAPIゲートウェイ Kong ご紹介
オープンソースのAPIゲートウェイ Kong ご紹介
briscola-tokyo
HDFSのスケーラビリティの限界を突破するためのさまざまな取り組み | Hadoop / Spark Conference Japan 2019 #hc...
HDFSのスケーラビリティの限界を突破するためのさまざまな取り組み | Hadoop / Spark Conference Japan 2019 #hc...
Yahoo!デベロッパーネットワーク
pgvectorを使ってChatGPTとPostgreSQLを連携してみよう!(PostgreSQL Conference Japan 2023 発表資料)
pgvectorを使ってChatGPTとPostgreSQLを連携してみよう!(PostgreSQL Conference Japan 2023 発表資料)
NTT DATA Technology & Innovation
KafkaとAWS Kinesisの比較
KafkaとAWS Kinesisの比較
Yoshiyasu SAEKI
これで怖くない!?大規模環境で体験するDB負荷対策~垂直から水平の彼方へ~
これで怖くない!?大規模環境で体験するDB負荷対策~垂直から水平の彼方へ~
hideakikabuto
分散ストレージソフトウェアCeph・アーキテクチャー概要
分散ストレージソフトウェアCeph・アーキテクチャー概要
Etsuji Nakai
CRX: Container Runtime Executive
CRX: Container Runtime Executive
imurata8203
分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)
NTT Communications Technology Development
What's hot
(20)
SPAセキュリティ入門~PHP Conference Japan 2021
SPAセキュリティ入門~PHP Conference Japan 2021
NGINXをBFF (Backend for Frontend)として利用した話
NGINXをBFF (Backend for Frontend)として利用した話
20190828 AWS Black Belt Online Seminar Amazon Aurora with PostgreSQL Compatib...
20190828 AWS Black Belt Online Seminar Amazon Aurora with PostgreSQL Compatib...
ブレソルでテラバイト級データのALTERを短時間で終わらせる
ブレソルでテラバイト級データのALTERを短時間で終わらせる
コンテナとimmutableとわたし。あとセキュリティ。(Kubernetes Novice Tokyo #15 発表資料)
コンテナとimmutableとわたし。あとセキュリティ。(Kubernetes Novice Tokyo #15 発表資料)
20190911 AWS Black Belt Online Seminar AWS Batch
20190911 AWS Black Belt Online Seminar AWS Batch
Oracle Database (CDB) on Docker を動かしてみる
Oracle Database (CDB) on Docker を動かしてみる
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
忙しい人の5分で分かるMesos入門 - Mesos って何だ?
忙しい人の5分で分かるMesos入門 - Mesos って何だ?
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
Java EE から Quarkus による開発への移行について
Java EE から Quarkus による開発への移行について
マイクロサービス化設計入門 - AWS Dev Day Tokyo 2017
マイクロサービス化設計入門 - AWS Dev Day Tokyo 2017
オープンソースのAPIゲートウェイ Kong ご紹介
オープンソースのAPIゲートウェイ Kong ご紹介
HDFSのスケーラビリティの限界を突破するためのさまざまな取り組み | Hadoop / Spark Conference Japan 2019 #hc...
HDFSのスケーラビリティの限界を突破するためのさまざまな取り組み | Hadoop / Spark Conference Japan 2019 #hc...
pgvectorを使ってChatGPTとPostgreSQLを連携してみよう!(PostgreSQL Conference Japan 2023 発表資料)
pgvectorを使ってChatGPTとPostgreSQLを連携してみよう!(PostgreSQL Conference Japan 2023 発表資料)
KafkaとAWS Kinesisの比較
KafkaとAWS Kinesisの比較
これで怖くない!?大規模環境で体験するDB負荷対策~垂直から水平の彼方へ~
これで怖くない!?大規模環境で体験するDB負荷対策~垂直から水平の彼方へ~
分散ストレージソフトウェアCeph・アーキテクチャー概要
分散ストレージソフトウェアCeph・アーキテクチャー概要
CRX: Container Runtime Executive
CRX: Container Runtime Executive
分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)
Viewers also liked
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Toshiaki Maki
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Toshiaki Maki
Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解
都元ダイスケ Miyamoto
Lineにおけるspring frameworkの活用
Lineにおけるspring frameworkの活用
Tokuhiro Matsuno
Spring Day 2016 springの現在過去未来
Spring Day 2016 springの現在過去未来
Yuichi Hasegawa
Spring bootで学ぶ初めてのwebアプリ開発
Spring bootで学ぶ初めてのwebアプリ開発
terahide
アメブロの大規模システム刷新と それを支えるSpring
アメブロの大規模システム刷新と それを支えるSpring
Takuya Hattori
Spring 5に備えるリアクティブプログラミング入門
Spring 5に備えるリアクティブプログラミング入門
Takuya Iwatsuka
Springを使ったwebアプリにリファクタリングしよう
Springを使ったwebアプリにリファクタリングしよう
土岐 孝平
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Yuichi Hasegawa
Grailsでドメイン駆動設計を実践する時の勘所
Grailsでドメイン駆動設計を実践する時の勘所
Takuma Watabiki
Cloud Foundry x Wagby
Cloud Foundry x Wagby
Yoshinori Nie
楽天トラベルとSpring(Spring Day 2016)
楽天トラベルとSpring(Spring Day 2016)
Rakuten Group, Inc.
ビッグじゃなくても使えるSpark Streaming
ビッグじゃなくても使えるSpark Streaming
chibochibo
Distributed Tracing Velocity2016
Distributed Tracing Velocity2016
Reshmi Krishna
Distributed tracing - get a grasp on your production
Distributed tracing - get a grasp on your production
nklmish
Spring CloudとZipkinを利用した分散トレーシング
Spring CloudとZipkinを利用した分散トレーシング
Rakuten Group, Inc.
形態素解析
形態素解析
Works Applications
WalB: Real-time and Incremental Backup System for Block Devices
WalB: Real-time and Incremental Backup System for Block Devices
uchan_nos
3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術
Ryo Mitoma
Viewers also liked
(20)
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解
Lineにおけるspring frameworkの活用
Lineにおけるspring frameworkの活用
Spring Day 2016 springの現在過去未来
Spring Day 2016 springの現在過去未来
Spring bootで学ぶ初めてのwebアプリ開発
Spring bootで学ぶ初めてのwebアプリ開発
アメブロの大規模システム刷新と それを支えるSpring
アメブロの大規模システム刷新と それを支えるSpring
Spring 5に備えるリアクティブプログラミング入門
Spring 5に備えるリアクティブプログラミング入門
Springを使ったwebアプリにリファクタリングしよう
Springを使ったwebアプリにリファクタリングしよう
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Grailsでドメイン駆動設計を実践する時の勘所
Grailsでドメイン駆動設計を実践する時の勘所
Cloud Foundry x Wagby
Cloud Foundry x Wagby
楽天トラベルとSpring(Spring Day 2016)
楽天トラベルとSpring(Spring Day 2016)
ビッグじゃなくても使えるSpark Streaming
ビッグじゃなくても使えるSpark Streaming
Distributed Tracing Velocity2016
Distributed Tracing Velocity2016
Distributed tracing - get a grasp on your production
Distributed tracing - get a grasp on your production
Spring CloudとZipkinを利用した分散トレーシング
Spring CloudとZipkinを利用した分散トレーシング
形態素解析
形態素解析
WalB: Real-time and Incremental Backup System for Block Devices
WalB: Real-time and Incremental Backup System for Block Devices
3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術
Similar to Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
Spring Cloud Servicesの紹介 #pcf_tokyo
Spring Cloud Servicesの紹介 #pcf_tokyo
Toshiaki Maki
Meetup: Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
Timothy Spann
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
Timothy Spann
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Timothy Spann
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Toshiaki Maki
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
VMware Tanzu
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Docker, Inc.
Hadoop application architectures - using Customer 360 as an example
Hadoop application architectures - using Customer 360 as an example
hadooparchbook
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
VMware Tanzu
Episode 3: Kubernetes and Big Data Services
Episode 3: Kubernetes and Big Data Services
Mesosphere Inc.
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
OSS On Azure
Big Data Analytics with Spark
Big Data Analytics with Spark
DataStax Academy
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Amazon Web Services
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
Kai Wähner
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Amazon Web Services
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Esther Vasiete
Fraud Detection for Israel BigThings Meetup
Fraud Detection for Israel BigThings Meetup
Gwen (Chen) Shapira
DevOps Days Tel Aviv - Serverless Architecture
DevOps Days Tel Aviv - Serverless Architecture
Antons Kranga
Spark Streaming with Azure Databricks
Spark Streaming with Azure Databricks
Dustin Vannoy
Spark Summit EU talk by Christos Erotocritou
Spark Summit EU talk by Christos Erotocritou
Spark Summit
Similar to Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
(20)
Spring Cloud Servicesの紹介 #pcf_tokyo
Spring Cloud Servicesの紹介 #pcf_tokyo
Meetup: Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Hadoop application architectures - using Customer 360 as an example
Hadoop application architectures - using Customer 360 as an example
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Episode 3: Kubernetes and Big Data Services
Episode 3: Kubernetes and Big Data Services
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Big Data Analytics with Spark
Big Data Analytics with Spark
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Fraud Detection for Israel BigThings Meetup
Fraud Detection for Israel BigThings Meetup
DevOps Days Tel Aviv - Serverless Architecture
DevOps Days Tel Aviv - Serverless Architecture
Spark Streaming with Azure Databricks
Spark Streaming with Azure Databricks
Spark Summit EU talk by Christos Erotocritou
Spark Summit EU talk by Christos Erotocritou
More from Toshiaki Maki
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
Toshiaki Maki
Concourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyo
Toshiaki Maki
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Toshiaki Maki
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
Toshiaki Maki
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Toshiaki Maki
Spring Boot Actuator 2.0 & Micrometer
Spring Boot Actuator 2.0 & Micrometer
Toshiaki Maki
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Toshiaki Maki
Spring Cloud Function & Project riff #jsug
Spring Cloud Function & Project riff #jsug
Toshiaki Maki
Introduction to Spring WebFlux #jsug #sf_a1
Introduction to Spring WebFlux #jsug #sf_a1
Toshiaki Maki
BOSH / CF Deployment in modern ways #cf_tokyo
BOSH / CF Deployment in modern ways #cf_tokyo
Toshiaki Maki
Why PCF is the best platform for Spring Boot
Why PCF is the best platform for Spring Boot
Toshiaki Maki
Zipkin Components #zipkin_jp
Zipkin Components #zipkin_jp
Toshiaki Maki
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
Toshiaki Maki
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Toshiaki Maki
Spring ❤️ Kotlin #jjug
Spring ❤️ Kotlin #jjug
Toshiaki Maki
Managing your Docker image continuously with Concourse CI
Managing your Docker image continuously with Concourse CI
Toshiaki Maki
Short Lived Tasks in Cloud Foundry #cfdtokyo
Short Lived Tasks in Cloud Foundry #cfdtokyo
Toshiaki Maki
今すぐ始めるCloud Foundry #hackt #hackt_k
今すぐ始めるCloud Foundry #hackt #hackt_k
Toshiaki Maki
Team Support in Concourse CI 2.0 #concourse_tokyo
Team Support in Concourse CI 2.0 #concourse_tokyo
Toshiaki Maki
From Zero to Hero with REST and OAuth2 #jjug
From Zero to Hero with REST and OAuth2 #jjug
Toshiaki Maki
More from Toshiaki Maki
(20)
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
Concourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyo
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer
Spring Boot Actuator 2.0 & Micrometer
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Spring Cloud Function & Project riff #jsug
Spring Cloud Function & Project riff #jsug
Introduction to Spring WebFlux #jsug #sf_a1
Introduction to Spring WebFlux #jsug #sf_a1
BOSH / CF Deployment in modern ways #cf_tokyo
BOSH / CF Deployment in modern ways #cf_tokyo
Why PCF is the best platform for Spring Boot
Why PCF is the best platform for Spring Boot
Zipkin Components #zipkin_jp
Zipkin Components #zipkin_jp
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Spring ❤️ Kotlin #jjug
Spring ❤️ Kotlin #jjug
Managing your Docker image continuously with Concourse CI
Managing your Docker image continuously with Concourse CI
Short Lived Tasks in Cloud Foundry #cfdtokyo
Short Lived Tasks in Cloud Foundry #cfdtokyo
今すぐ始めるCloud Foundry #hackt #hackt_k
今すぐ始めるCloud Foundry #hackt #hackt_k
Team Support in Concourse CI 2.0 #concourse_tokyo
Team Support in Concourse CI 2.0 #concourse_tokyo
From Zero to Hero with REST and OAuth2 #jjug
From Zero to Hero with REST and OAuth2 #jjug
Recently uploaded
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Katpro Technologies
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
sudhanshuwaghmare1
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
Antenna Manufacturer Coco
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
UK Journal
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Igalia
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
The Digital Insurer
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
wesley chun
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Enterprise Knowledge
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Gabriella Davis
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
RTylerCroy
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
HampshireHUG
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Neo4j
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Principled Technologies
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
Khem
Slack Application Development 101 Slides
Slack Application Development 101 Slides
praypatel2
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Safe Software
Recently uploaded
(20)
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
Slack Application Development 101 Slides
Slack Application Development 101 Slides
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
1.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices with Spring Cloud Stream, Task, and Data Flow Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug
2.
© 2016 Pivotal
Software, Inc. All rights reserved. Who am I ? • Toshiaki Maki (@making) http://blog.ik.am • Sr. Solutions Architect @Pivotal • Spring Framework enthusiast bit.ly/hajiboot2
3.
© 2016 Pivotal
Software, Inc. All rights reserved. Input Motivation HTTP TCP S3 JDBC Rabbit JMS Twitter Syslog Output HDFS Cassandra HAWQ Greenplum JDBC S3 TCP Gemfire ❓❔❓ • Real Time Analysis • Data Ingestion • ETL Process
4.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD (eXtreme Data)
5.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD (eXtreme Data) 🤔
6.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
7.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs 😇
8.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
9.
© 2016 Pivotal
Software, Inc. All rights reserved. Modern Platforms Cloud Foundry YARN Kubernetes Mesos Docker Swarm Nomad OpenShift
10.
© 2016 Pivotal
Software, Inc. All rights reserved. Cloud Native Redesign •From multiple modules embedded in a container to standalone executable applications •From our own runtime to delegating to existing modern platforms
11.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
12.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
13.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
14.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
15.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
16.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
17.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
18.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
19.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
20.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications
21.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream
22.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task
23.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task Orchestration Layer
24.
© 2016 Pivotal
Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch
25.
© 2016 Pivotal
Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch • Spring Cloud Deployer Local • Spring Cloud Deployer Cloud Foundry • Spring Cloud Deployer Yarn • Spring Cloud Deployer Kubernetes • Spring Cloud Deployer Mesos
26.
© 2016 Pivotal
Software, Inc. All rights reserved. SCDF Deployment Platform Spring Cloud Data Flow Server Deployer SPI REST API SCDFShell SCDF Flo
27.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
28.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
29.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream
30.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components (Spring Boot / Spring Integration) • Opinionated primitives for streaming applications • Persistent Pub/Sub • Consumer Groups • Partitioning Support • Pluggable messaging middleware bindings source | processor | sink
31.
© 2016 Pivotal
Software, Inc. All rights reserved. JJUG CCC in 2 weeks
32.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest
33.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest Twitter Stream Cassandraingest
34.
© 2016 Pivotal
Software, Inc. All rights reserved. Message Binders • @EnableBinding • Binder Implementations • Production-Ready • Rabbit MQ • Apache Kafka • Experimental • JMS • Google PubSub
35.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Sink) @SpringBootApplication @EnableBinding(Sink.class) public class DemoSinkApp { @StreamListener(Sink.INPUT) void receive(Message<String> message) { System.out.println("Received " + message); } public static void main(String[] args) { SpringApplication.run(DemoSinkApp.class, args); } }
36.
© 2016 Pivotal
Software, Inc. All rights reserved. Sink Properties (Sink) spring.cloud.stream.bindings.input.destination=demo-strm demo- strm input
37.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class) public class DemoSourceApp { @Autowired Source source; @GetMapping void send(@RequestParam String text) { source.output() .send(MessageBuilder.withPayload(text).build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); } }
38.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class) public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() { return () -> MessageBuilder.withPayload("Hi").build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); }}
39.
© 2016 Pivotal
Software, Inc. All rights reserved. Properties (Source) spring.cloud.stream.bindings.output.destination=demo-strm demo- strm Source output
40.
© 2016 Pivotal
Software, Inc. All rights reserved. Binder (RabbitMQ) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId> </dependency> demo- strm
41.
© 2016 Pivotal
Software, Inc. All rights reserved. Binder (Apache Kafka) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-kafka</artifactId> </dependency> demo- strm
42.
© 2016 Pivotal
Software, Inc. All rights reserved. Sink Pipeline demo- strm input Source output source | sink
43.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Processor) @SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT) void receive(String text) { return "[[" + text + "]]"; } public static void main(String[] args) { SpringApplication.run(DemoProcessorApp.class, args); } }
44.
© 2016 Pivotal
Software, Inc. All rights reserved. Properties (Processor) spring.cloud.stream.bindings.output.destination=my-source spring.cloud.stream.bindings.input.destination=my-source spring.cloud.stream.bindings.output.destination=my-proc spring.cloud.stream.bindings.input.destination=my-proc Source Processor Sink
45.
© 2016 Pivotal
Software, Inc. All rights reserved. Pipeline my- source Source output source | processor | sink Processor output input my-proc Sink input
46.
© 2016 Pivotal
Software, Inc. All rights reserved. Reactive API Support @SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT) public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) { return stream.map(text -> "[[" + text + "]]"); } public static void main(String[] args) { SpringApplication.run(DemoProcessorRxApp.class, args); } }
47.
© 2016 Pivotal
Software, Inc. All rights reserved. Reactive API Support @StreamListener @Output(Processor.OUTPUT) public Flux<AverageData> receive(@Input(Processor.INPUT) Flux<SensorData> stream) { return stream.window(Duration.ofSecond(20), Duration.ofSecond(10)) .flatMap(win -> win.groupBy(sensor -> sensor.id)) .flatMap(group -> calcAverage(group)); }
48.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream App Starters Source Processor Sink file ftp gemfire gemfire-cq http jdbc jms load-generator loggregator mail mongodb rabbit s3 sftp syslog tcp tcp-client time trigger triggertask twitterstream bridge filter groovy-filter groovy- transform httpclient pommel scriptable- transform splitter tcp-client transform aggregate- counter cassandra counter field-value- counter file ftp gemfire gpfdist hfs hdfs-dataset jdbc log rabbit redis-pubsub router s3 sftp task-launcher- local task-lancher- yarn tcp throughput web socket http://cloud.spring.io/spring-cloud-stream-app-starters/
49.
© 2016 Pivotal
Software, Inc. All rights reserved. Stream Orchestration in SCDF ingest = twitterstream | cassandra dataflow:> stream create --name=ingest --definition="twitterstream | cassandra" dataflow:> stream deploy --name=ingest |
50.
© 2016 Pivotal
Software, Inc. All rights reserved. Import apps dataflow:> app import --uri http://bit.ly/1-0-4- GA-stream-applications-rabbit-maven rabbit maven kafka docker stream-applications- http://cloud.spring.io/spring-cloud-stream-app-starters/
51.
© 2016 Pivotal
Software, Inc. All rights reserved. Register your own apps dataflow:> app register --name foo --type source --uri http://example.com/foo-source-1.0.jar dataflow:> app register --name foo --type source --uri maven://com.example:foo-source:1.0 dataflow:> app register --name foo --type source --uri docker:myapps/foo-source:1.0 dataflow:> app register --name foo --type source --uri file:///tmp/foo-source:1.0
52.
© 2016 Pivotal
Software, Inc. All rights reserved. Instance Count dataflow:> stream create --name=s1 --definition="http | work | hdfs" dataflow:> stream deploy --name=s1 -- properties="app.http.count=2,app.work.count=3,app .hdfs.count=4"
53.
© 2016 Pivotal
Software, Inc. All rights reserved. Instance Count http work hdfs http work work hdfs hdfs hdfs LoadBalancer
54.
© 2016 Pivotal
Software, Inc. All rights reserved. Resource Management dataflow:> stream deploy --name=s1 -- properties="app.work.spring.cloud.deployer.cloudf oundry.memory=2048"
55.
© 2016 Pivotal
Software, Inc. All rights reserved. Resource Management http http work work work hdfs hdfs hdfs hdfs
56.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Flo http://projects.spring.io/spring-flo/
57.
© 2016 Pivotal
Software, Inc. All rights reserved. Stream Core Features •Persistent Pub-Sub •Consumer Group •Partitioning Support
58.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average Top Ns1.http s1.ave Message Broker
59.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top Ns1.http s1.ave Message Broker
60.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker
61.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}
62.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38}
63.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
64.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature": {"id":1, "temperature":38} {"id":1, "temperature":38}
65.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS
66.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} Average Average HDFS HDFS
67.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38} Average Average HDFS HDFS
68.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
69.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩
70.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 Consumer Group
71.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 spring.cloud.stream.bindings.<channelName>.group=ave Spring Cloud Stream group is configured by default 😁 Spring Cloud Data Flow Consumer Group
72.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs
73.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}
74.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}{"id":1, "temperature":38}
75.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38}
76.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓
77.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓consumer group subscriptions are durable 😁
78.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average
79.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
80.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
81.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
82.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
83.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
84.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩
85.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩 Partitioning Support(Stateful Stream)
86.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩stream deploy --name=xxxx --properties= "app.http.producer.partitionKeyExpression=payload.id" spring.cloud.stream.bindings.<channelName>.producer.par titionKeyExpression=payload.id Spring Cloud Stream Spring Cloud Data Flow Partitioning Support(Stateful Stream)
87.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average
88.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
89.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
90.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
91.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
92.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
93.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 🤓
94.
© 2016 Pivotal
Software, Inc. All rights reserved. Simple Real Time Analytics tweets = twitterstream | hdfs analytics = :ingest.twitterstream > field-value-counter --fieldName=lang HTTP s1.http HDFS COUNTER Data Flow Server REST API
95.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Demo (Twitter Stream!)
96.
© 2016 Pivotal
Software, Inc. All rights reserved.
97.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Task
98.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Task • Spring Boot based framework for short lived task • Spring Boot's CommandLineRunner or Spring Batch • Result of each process persists in Task Repository • Well Integrated with Spring Batch (Job Repository)
99.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (CLR) @SpringBootApplication @EnableTask public class DemoTaskApp { @Bean CommandLineRunner clr() { return args -> System.out.println("Task!"); } public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); } } spring.application.name=hello
100.
© 2016 Pivotal
Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello 3 Tue Nov 15 16:49:08 JST 2016 Tue Nov 15 16:49:08 JST 2016 0 hello 2 Tue Nov 15 16:48:54 JST 2016 Tue Nov 15 16:48:55 JST 2016 0 hello 1 Tue Nov 15 16:48:23 JST 2016 Tue Nov 15 16:48:23 JST 2016 0
101.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Spring Batch) @SpringBootApplication @EnableBatchProcessing @EnableTask public class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory; @Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()} public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); }} https://github.com/making/cf-spring-batch-demo spring.application.name=hello-batch
102.
© 2016 Pivotal
Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello- batch 5 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 0 hello- batch 4 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 0
103.
© 2016 Pivotal
Software, Inc. All rights reserved. Job Execution List ID Task ID Start Time Step Execution Count Definition Status 2 5 Tue Nov 15 17:28:49 JST 2016 2 Created 1 4 Tue Nov 15 17:27:56 JST 2016 2 Created
104.
© 2016 Pivotal
Software, Inc. All rights reserved. Step Execution List ID Step Name Job Exec Id Start Time End Time Status 1 step1 1 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 COMPLETED 2 step2 1 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 COMPLETED
105.
© 2016 Pivotal
Software, Inc. All rights reserved. Task Orchestration in SCDF >task create hello --definition="timestamp --format=¥"yyyy¥"" >task launch hello timestamp Data Flow Server DB Task Name Start Time End Time Exit Code Exit Message Last Updated Time Parameters Message Broker
106.
© 2016 Pivotal
Software, Inc. All rights reserved. Task Execution List
107.
© 2016 Pivotal
Software, Inc. All rights reserved. Job Execution List
108.
© 2016 Pivotal
Software, Inc. All rights reserved. Job Execution Details
109.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow -> Task http | tasklaunchrequest --uri=... | task-lancher timestamp Data Flow Server DB Message Broker
110.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Demo (Scalable Tasks)
111.
© 2016 Pivotal
Software, Inc. All rights reserved. Pivotal Cloud Foundry PDF PDF PDF PDF S3 Spring Cloud Data Flow Task Task Task Task PDF PDF Spring Cloud Task Spring Batch index index source sink
112.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Getting Started with SCDF on Cloud Foundry
113.
© 2016 Pivotal
Software, Inc. All rights reserved. Install PCF Dev • https://docs.pivotal.io/pcf-dev • Cloud Foundry on your laptop • Included • Redis / RabbitMQ / MySQL • Spring Cloud Services • Install with cf dev start
114.
© 2016 Pivotal
Software, Inc. All rights reserved. cf create-service p-mysql 512mb df-mysql cf create-service p-rabbitmq standard df-rebbitmq cf create-service p-redis shared-vm df-redis cf push dataflow-server -p spring-cloud-dataflow- server-cloudfoundry-1.0.1.RELEASE.jar Deploy SCDF Server on PCF Dev
115.
© 2016 Pivotal
Software, Inc. All rights reserved. Tutorial • 📗 https://blog.ik.am/entries/396 • 💎 https://github.com/making-demo-scdf/spring-cloud- dataflow-cookbook
116.
© 2016 Pivotal
Software, Inc. All rights reserved. Upcoming Features • Some ‘porting’ from XD • Batch Job DSL + Designer • Role based access • Looking forward • Spring Cloud Sleuth • JavaDSL • In-place application version upgrades with Spinnaker • Application Groups • Polyglot • Expanded analytics with Redis and Python/R ecosystem • More provided apps/tasks •SCDF Tile for Pivotal Cloud Foundry
117.
© 2016 Pivotal
Software, Inc. All rights reserved. Thanks! • References • http://www.slideshare.net/SpringCentral/data-microservices-in-t cloud ✨ • https://cloud.spring.io/spring-cloud-dataflow/ • http://cloud.spring.io/spring-cloud-stream/ • http://cloud.spring.io/spring-cloud-stream-app-starters/ • https://github.com/spring-cloud/spring-cloud-task/ • http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry
Download now