データの民主化のために StackStorm を活用した事例
•
4 likes•3,808 views
Tech Night @ Shiodome #8 https://techsio.connpass.com/event/83273/ 発表資料
Recommended
More Related Content
What's hot
What's hot (20)
Similar to データの民主化のために StackStorm を活用した事例
Similar to データの民主化のために StackStorm を活用した事例 (20)
Recently uploaded
Recently uploaded (20)
データの民主化のために StackStorm を活用した事例
- 1. StackStorm / @laclefyoshi / ysaeki@r.recruit.co.jp
- 3. : / @laclefyoshi • 2011/04 • 2015/09 • • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) • Apache Kafka 0.11 Exactly Once Semantics (Apache Kafka Meetup Japan #3; 2017) • Apache Kafka (Apache Kafka Meetup Japan #4; 2018) 3
- 4. StackStorm
- 5. StackStorm • https://stackstorm.com/ • OSS + LDAP Auth plugin https://git.io/vxbbJ • • Shell Script • Crontab 5
- 6. • JP1: • Digdag: • Jenkins: CI • Airflow: GCP Cloud Composer • Oozie: Hadoop 6
- 7. StackStorm • • High Availability • Crontab Scheduler • • • • Runner Action Workflow • • REST API 7
- 8. StackStorm API 1. test.test Action 2. API pack: test name: test runner_type: remote-shell-cmd parameters: p: type: string required: true curl -k -X POST -H "Content-type: application/json" -H "X-Auth-Token: $TOKEN" --data-binary '{"action": “test.test", “parameters": {"p": "..."}}' "https://${ST2_ADDR}/api/v1/executions" 8
- 9. StackStorm • Design UI Commercial • GUI • OSS • Commercial Role Based Access Control • OSS • Action Workflow • Jinja/YAQL 9
- 10. st2XXX, mistral-XXX replication cluster mistral (PostgreSQL 9.6) Runner Action Workflow Rule 10
- 11. • • 1 • Timer • st2rulesengine Timer HA • https://docs.stackstorm.com/reference/ha.html#st2rulesengine • MongoDB Primary st2rulesengine Timer • Concurrency policy Zookeeper Timer 11
- 12. GCP? • BigQuery • BigQuery • SQL • Tableau • Pandas R • StackStorm BigQuery 12
- 15. 1/2 • 15
- 17. StackStorm 1. StackStorm 2. 3. StackStorm 4. StackStorm 5. 6. 17
- 18. StackStorm • Amazon Data Value Chain Subscribe Deliver https://www.slideshare.net/AmazonWebServices/a-look-under-the-hood-how- amazoncom-uses-aws-services-for-analytics-at-massive-scale-abd329-reinvent-2017 18
- 19. Action • Action • • • • LDAP • Stackdriver • Action • SQL BigQuery • BigQuery CSV • 19
- 20. • StackStorm Action • Action Workflow • YAML • • YAML • JP1 GUI 20
- 21. 21
- 22. API • YAML • GUI API • Action Workflow • StackStorm Action Workflow API • API • 22
- 23. 1/2 23 2. StackStorm Web UI StackStorm API 1. Action Workflow API
- 24. 2/2 24 JP1 2. JP1 1 StackStorm API 1. Action Workflow API
- 25. • StackStorm • API • StackStorm • • Rule • JP1 • Workflow • 25
- 26. 26 • StackStorm • Action Workflow YAML • • Action Workflow