Submit Search
Upload
StackStormを1年間データ基盤で使ってみてぶつかったトラブルとその解決策の共有
•
2 likes
•
1,357 views
Yoshiyasu SAEKI
Follow
StackStorm Meetup JP #7 https://stackstorm-jp.connpass.com/event/125944/ 発表資料
Read less
Read more
Technology
Report
Share
Report
Share
1 of 16
Download now
Download to read offline
Recommended
グラフデータベース Neptune 使ってみた
グラフデータベース Neptune 使ってみた
Yoshiyasu SAEKI
データの民主化のために StackStorm を活用した事例
データの民主化のために StackStorm を活用した事例
Yoshiyasu SAEKI
Apache Kafkaとグラフデータベースによる成長するネットワークグラフを分析・可視化する基盤
Apache Kafkaとグラフデータベースによる成長するネットワークグラフを分析・可視化する基盤
Yoshiyasu SAEKI
Spark Streamingによるリアルタイムユーザ属性推定
Spark Streamingによるリアルタイムユーザ属性推定
Yoshiyasu SAEKI
ストリーミングデータのアドホック分析エンジンの比較
ストリーミングデータのアドホック分析エンジンの比較
Yoshiyasu SAEKI
Queryable State for Kafka Streamsを使ってみた
Queryable State for Kafka Streamsを使ってみた
Yoshiyasu SAEKI
Voldemortの紹介
Voldemortの紹介
Yoshiyasu SAEKI
Apache Kafka 0.11 の Exactly Once Semantics
Apache Kafka 0.11 の Exactly Once Semantics
Yoshiyasu SAEKI
Recommended
グラフデータベース Neptune 使ってみた
グラフデータベース Neptune 使ってみた
Yoshiyasu SAEKI
データの民主化のために StackStorm を活用した事例
データの民主化のために StackStorm を活用した事例
Yoshiyasu SAEKI
Apache Kafkaとグラフデータベースによる成長するネットワークグラフを分析・可視化する基盤
Apache Kafkaとグラフデータベースによる成長するネットワークグラフを分析・可視化する基盤
Yoshiyasu SAEKI
Spark Streamingによるリアルタイムユーザ属性推定
Spark Streamingによるリアルタイムユーザ属性推定
Yoshiyasu SAEKI
ストリーミングデータのアドホック分析エンジンの比較
ストリーミングデータのアドホック分析エンジンの比較
Yoshiyasu SAEKI
Queryable State for Kafka Streamsを使ってみた
Queryable State for Kafka Streamsを使ってみた
Yoshiyasu SAEKI
Voldemortの紹介
Voldemortの紹介
Yoshiyasu SAEKI
Apache Kafka 0.11 の Exactly Once Semantics
Apache Kafka 0.11 の Exactly Once Semantics
Yoshiyasu SAEKI
Apache Sparkにおけるメモリ - アプリケーションを落とさないメモリ設計手法 -
Apache Sparkにおけるメモリ - アプリケーションを落とさないメモリ設計手法 -
Yoshiyasu SAEKI
Railsで作るBFFの功罪
Railsで作るBFFの功罪
Recruit Lifestyle Co., Ltd.
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
台灣資料科學年會
Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafka
Timothy Spann
Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...
Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...
Matt Stubbs
Presto Strata Hadoop SJ 2016 short talk
Presto Strata Hadoop SJ 2016 short talk
kbajda
ELK at LinkedIn - Kafka, scaling, lessons learned
ELK at LinkedIn - Kafka, scaling, lessons learned
Tin Le
Log analysis using Logstash,ElasticSearch and Kibana
Log analysis using Logstash,ElasticSearch and Kibana
Avinash Ramineni
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
HostedbyConfluent
PWL: One VM to Rule Them All
PWL: One VM to Rule Them All
Aysylu Greenberg
Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...
Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...
confluent
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Cohesive Networks
ELK introduction
ELK introduction
Waldemar Neto
Kafka Summit SF 2017 - Streaming Processing in Python – 10 ways to avoid summ...
Kafka Summit SF 2017 - Streaming Processing in Python – 10 ways to avoid summ...
confluent
Facebook Presto presentation
Facebook Presto presentation
Cyanny LIANG
E2E Data Pipeline - Apache Spark/Airflow/Livy
E2E Data Pipeline - Apache Spark/Airflow/Livy
Rikin Tanna
Apache Pulsar Community-Jennifer
Apache Pulsar Community-Jennifer
StreamNative
Api world apache nifi 101
Api world apache nifi 101
Timothy Spann
ApacheCon 2021: Apache NiFi 101- introduction and best practices
ApacheCon 2021: Apache NiFi 101- introduction and best practices
Timothy Spann
Log aggregation and analysis
Log aggregation and analysis
Dhaval Mehta
Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...
Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...
Jacek Laskowski
Perl ウェブ開発の中世〜CGI と Plack の間〜
Perl ウェブ開発の中世〜CGI と Plack の間〜
鉄次 尾形
More Related Content
What's hot
Apache Sparkにおけるメモリ - アプリケーションを落とさないメモリ設計手法 -
Apache Sparkにおけるメモリ - アプリケーションを落とさないメモリ設計手法 -
Yoshiyasu SAEKI
Railsで作るBFFの功罪
Railsで作るBFFの功罪
Recruit Lifestyle Co., Ltd.
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
台灣資料科學年會
Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafka
Timothy Spann
Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...
Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...
Matt Stubbs
Presto Strata Hadoop SJ 2016 short talk
Presto Strata Hadoop SJ 2016 short talk
kbajda
ELK at LinkedIn - Kafka, scaling, lessons learned
ELK at LinkedIn - Kafka, scaling, lessons learned
Tin Le
Log analysis using Logstash,ElasticSearch and Kibana
Log analysis using Logstash,ElasticSearch and Kibana
Avinash Ramineni
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
HostedbyConfluent
PWL: One VM to Rule Them All
PWL: One VM to Rule Them All
Aysylu Greenberg
Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...
Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...
confluent
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Cohesive Networks
ELK introduction
ELK introduction
Waldemar Neto
Kafka Summit SF 2017 - Streaming Processing in Python – 10 ways to avoid summ...
Kafka Summit SF 2017 - Streaming Processing in Python – 10 ways to avoid summ...
confluent
Facebook Presto presentation
Facebook Presto presentation
Cyanny LIANG
E2E Data Pipeline - Apache Spark/Airflow/Livy
E2E Data Pipeline - Apache Spark/Airflow/Livy
Rikin Tanna
Apache Pulsar Community-Jennifer
Apache Pulsar Community-Jennifer
StreamNative
Api world apache nifi 101
Api world apache nifi 101
Timothy Spann
ApacheCon 2021: Apache NiFi 101- introduction and best practices
ApacheCon 2021: Apache NiFi 101- introduction and best practices
Timothy Spann
Log aggregation and analysis
Log aggregation and analysis
Dhaval Mehta
What's hot
(20)
Apache Sparkにおけるメモリ - アプリケーションを落とさないメモリ設計手法 -
Apache Sparkにおけるメモリ - アプリケーションを落とさないメモリ設計手法 -
Railsで作るBFFの功罪
Railsで作るBFFの功罪
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
使用 Elasticsearch 及 Kibana 進行巨量資料搜尋及視覺化-曾書庭
Real time stock processing with apache nifi, apache flink and apache kafka
Real time stock processing with apache nifi, apache flink and apache kafka
Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...
Speed Up Your Apache Cassandra™ Applications: A Practical Guide to Reactive P...
Presto Strata Hadoop SJ 2016 short talk
Presto Strata Hadoop SJ 2016 short talk
ELK at LinkedIn - Kafka, scaling, lessons learned
ELK at LinkedIn - Kafka, scaling, lessons learned
Log analysis using Logstash,ElasticSearch and Kibana
Log analysis using Logstash,ElasticSearch and Kibana
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
Practical tips and tricks for Apache Kafka messages integration | Francesco T...
PWL: One VM to Rule Them All
PWL: One VM to Rule Them All
Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...
Building a newsfeed from the Universe: Data streams in astronomy (Maria Patte...
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
Lessons Learned in Deploying the ELK Stack (Elasticsearch, Logstash, and Kibana)
ELK introduction
ELK introduction
Kafka Summit SF 2017 - Streaming Processing in Python – 10 ways to avoid summ...
Kafka Summit SF 2017 - Streaming Processing in Python – 10 ways to avoid summ...
Facebook Presto presentation
Facebook Presto presentation
E2E Data Pipeline - Apache Spark/Airflow/Livy
E2E Data Pipeline - Apache Spark/Airflow/Livy
Apache Pulsar Community-Jennifer
Apache Pulsar Community-Jennifer
Api world apache nifi 101
Api world apache nifi 101
ApacheCon 2021: Apache NiFi 101- introduction and best practices
ApacheCon 2021: Apache NiFi 101- introduction and best practices
Log aggregation and analysis
Log aggregation and analysis
Similar to StackStormを1年間データ基盤で使ってみてぶつかったトラブルとその解決策の共有
Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...
Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...
Jacek Laskowski
Perl ウェブ開発の中世〜CGI と Plack の間〜
Perl ウェブ開発の中世〜CGI と Plack の間〜
鉄次 尾形
15年前に作ったアプリを現在に蘇らせてみた話
15年前に作ったアプリを現在に蘇らせてみた話
Naoki Nagazumi
Railsチュートリアルの歩き方 (第4版)
Railsチュートリアルの歩き方 (第4版)
Yohei Yasukawa
Making the big data ecosystem work together with Python & Apache Arrow, Apach...
Making the big data ecosystem work together with Python & Apache Arrow, Apach...
Holden Karau
Making the big data ecosystem work together with python apache arrow, spark,...
Making the big data ecosystem work together with python apache arrow, spark,...
Holden Karau
Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling
Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling
Databricks
Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...
Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...
Chris Fregly
DataStax & O'Reilly Media: Large Scale Data Analytics with Spark and Cassandr...
DataStax & O'Reilly Media: Large Scale Data Analytics with Spark and Cassandr...
DataStax Academy
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Helena Edelson
Exploring the Scala ecosystem
Exploring the Scala ecosystem
Denis
HPBigData2015 PSTL kafka spark vertica
HPBigData2015 PSTL kafka spark vertica
Jack Gudenkauf
Extending Spark SQL 2.4 with New Data Sources (Live Coding Session)
Extending Spark SQL 2.4 with New Data Sources (Live Coding Session)
Databricks
LLPlanets Lightning Talk Presentation No.8
LLPlanets Lightning Talk Presentation No.8
Yoji TAKEUCHI
Flink September 2015 Community Update
Flink September 2015 Community Update
Robert Metzger
PyCon JP 2011 Lightning Talk No.10
PyCon JP 2011 Lightning Talk No.10
Yoji TAKEUCHI
Hadoopsummit16 myui
Hadoopsummit16 myui
Makoto Yui
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Sarah Guido
20180417 hivemall meetup#4
20180417 hivemall meetup#4
Takeshi Yamamuro
Apereo CAS: State of the Project 2018
Apereo CAS: State of the Project 2018
Misagh Moayyed
Similar to StackStormを1年間データ基盤で使ってみてぶつかったトラブルとその解決策の共有
(20)
Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...
Kafka Streams VS Spark Structured Streaming - Modern Stream Processing Engin...
Perl ウェブ開発の中世〜CGI と Plack の間〜
Perl ウェブ開発の中世〜CGI と Plack の間〜
15年前に作ったアプリを現在に蘇らせてみた話
15年前に作ったアプリを現在に蘇らせてみた話
Railsチュートリアルの歩き方 (第4版)
Railsチュートリアルの歩き方 (第4版)
Making the big data ecosystem work together with Python & Apache Arrow, Apach...
Making the big data ecosystem work together with Python & Apache Arrow, Apach...
Making the big data ecosystem work together with python apache arrow, spark,...
Making the big data ecosystem work together with python apache arrow, spark,...
Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling
Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling
Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...
Kafka Summit SF Apr 26 2016 - Generating Real-time Recommendations with NiFi,...
DataStax & O'Reilly Media: Large Scale Data Analytics with Spark and Cassandr...
DataStax & O'Reilly Media: Large Scale Data Analytics with Spark and Cassandr...
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Exploring the Scala ecosystem
Exploring the Scala ecosystem
HPBigData2015 PSTL kafka spark vertica
HPBigData2015 PSTL kafka spark vertica
Extending Spark SQL 2.4 with New Data Sources (Live Coding Session)
Extending Spark SQL 2.4 with New Data Sources (Live Coding Session)
LLPlanets Lightning Talk Presentation No.8
LLPlanets Lightning Talk Presentation No.8
Flink September 2015 Community Update
Flink September 2015 Community Update
PyCon JP 2011 Lightning Talk No.10
PyCon JP 2011 Lightning Talk No.10
Hadoopsummit16 myui
Hadoopsummit16 myui
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at Bitly
20180417 hivemall meetup#4
20180417 hivemall meetup#4
Apereo CAS: State of the Project 2018
Apereo CAS: State of the Project 2018
Recently uploaded
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
LoriGlavin3
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
Bernd Ruecker
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
LoriGlavin3
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Pim van der Noll
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
Neo4j
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
Curtis Poe
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Nikki Chapple
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
Nicole Novielli
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
Kari Kakkonen
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
Wes McKinney
A Framework for Development in the AI Age
A Framework for Development in the AI Age
Cprime
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
LoriGlavin3
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
Knoldus Inc.
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
panagenda
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
Inflectra
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
itnewsafrica
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
Nathaniel Shimoni
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
LoriGlavin3
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
itnewsafrica
Recently uploaded
(20)
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
A Framework for Development in the AI Age
A Framework for Development in the AI Age
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
Data governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
StackStormを1年間データ基盤で使ってみてぶつかったトラブルとその解決策の共有
1.
StackStorm 1 / @laclefyoshi
/ ysaeki@r.recruit.co.jp
2.
: / @laclefyoshi •
2011/04 • 2015/09 • • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) • Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017) • Apache Kafka (Apache Kafka Meetup Japan #4; 2018) • StackStorm (Tech Night @ Shiodome #8; 2018) • Neptune (BigData-JAWS #13; 2018) 2
3.
StackStorm 1
4.
1 https://www.slideshare.net/laclefyoshi/stackstorm-99437154 4
5.
5
6.
logging.conf consoleHandler →journald st2xxx →
journald → rsyslogd 6
7.
Timeout 1/2 • 1 •
grpc : https://github.com/grpc/grpc/issues/17986 • St2 gsutil • gsutil v1.9.0 St2 ↓ St2 Status = Timeout 7
8.
Timeout 2/2 • 2 • •
AM3 logrotate logrotate ↓ St2 Status = Timeout 8
9.
Running ↓ St2 Status =
Running • • logrotate etc. st2actionrunner RabbitMQ PostgreSQL for Mistral 9
10.
CronTimer • st2ruleengine CronTimer 10
11.
• st2 get
… ASCII Python2 : st2 execution get … -d DEBUG st2 execution get … -j JSON CLI st2 execution get … UnicodeEncodeError: 'ascii' codec can't encode characters 11
12.
Web UI • Web
UI • React Run Web UI CLI Web UI ↓ 12
13.
1/2 • 1 stdbuf -o0
1 core.remote 2 ↓ 2 1 KB 10KB 13
14.
2/2 • pseudo tty
SSH • st2common/st2common/runners/paramiko_ssh.py sudo paramiko.channel.Channel.get_pty 370 get_pty 1 core.remote 2 ↓ 2 1 KB 10KB 14
15.
16.
• St2 React
OS • OSS • v2.9 • v3 Python3 • Orquesta Mistral 16
Download now