Embulk Meetup #3 https://techplay.jp/event/618872

1. Embulk Meetup #3
My
Use case

2. Embulk and
Machine Learning
Infrastructure

3. self‐introduction
• @toyama0919
• Planning and Development Division.
• Analytics infra.
• Embulk using one and a half years.
• Machine learning experience is one year.

4. Own plugins
• embulk-filter-kuromoji
• embulk-input-presto
• embulk-parser-ltsv
• embulk-filter-split
• embulk-filter-icu4j
• embulk-input-elasticsearch
• embulk-filter-woothee
• …. Total 19 plugins.

5. situation

6. Our development situation
• We love OSS.
• We have a robust analytic environment.
• We love SQL.
• We have corporate culture that can
challenge new technologies.

7. Ruby Batch
CollectBatchVisualize Data Store
Write Read

8. Our business situation

9. Our business situation
• We manage and operate web site of BtoB.
• Our data lifecycle is long.
• Business side not write sql.
• watching re:dash and Adobe analytics.

10. Embulk our use case
• Create point in time data.
• Create machine learning data.
• Execute Machine Learning on the Cloud.
• Natural language processing.
• Marketing Automation.
• One shot data migration.
• Large scale parallel crawl.
• etc…

11. OUTPUTINPUT
http
FILTER
ICU
kuromoji
ROW
ruby_proc

12. Natural language
processing

13. NLP use case
• Product category classification.
• Normalization of notation fluctuation
• Morphological analysis.

14. NLP plugins
• embulk-filter-kuromoji.
• embulk-filter-icu4j
• filter-regex-devide(closed)
• filter-aggregate_synonym(closed)
• Other closed 20 plugins…

15. OUTPUTFILTERINPUT
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
F
I
L
T
E
R
①
②
③

16. Digdag my use case
• Digdag is very easy to use workflow
engine.
• Solving complex dependencies of
machine learning and natural language
processing.
• Crawl web site.

17. embulk-filter-kuromoji
• Kuromoji is an open source Japanese
morphological analyzer written in Java.
• morphological analyze on Embulk.
• Support Neologd.
• Support custom dictionary.
• 安全と安心は違う => [“安全”,”と”,“安
心”,“は”,“違う”]

18. embulk-filter-icu4j
• icu is Unicode normalize library.
• Unicode normalize on Embulk.
• https://hondou.homedns.org/pukiwiki/pukiwi
ki.php?Java%2520ICU4J.
• Many normalize pettern.
– Any-Katakana
– Any-Lower
– Any-NFC
– Any-NFKC

19. Classic Machine
Learning

20. Amazon Machine Learning
• Only classical machine learning support.
– Multiclass Classification
– Binary Classification
– Logistic regression
• Not DeepLearning.
• collaborate seamlessly with our data.
– Our all data on AWS.

21. AML Use Case
• Detection of incomplete data.
• Marketing automation
– During verification…
– Filtering noise data.
– Market scoring.

22. PredictionData Source
Amazon Machine Learning
Ruby
READ WRITE
Call API
1
2

23. Machine Learning
API

24. DeepLearning Cloud vendor
• Amazon AI(Amazon)
• Cloud Machine Learning(Google)
• Cognitive services(Microsoft)
• Watson(IBM)
• DataRobot

25. DL as a service type
• Image recognition.
• Speech recognition.
• Natural Language Processing.
• Machine translation.

26. Image recognition
• Amazon Rekognition
• Google Cloud Vision API
• Azure Computer Vision API

27. Image recognition
AWS GCP Azure
Label Detection ◎ ◎ ◎
Explicit Content Detection ◯ ◯ ◯
Logo Detection ☓ ◯ ☓
Landmark Detection ☓ ◯ ☓
Text Detection ☓ ◎ ◯
Face Detection ◯ ◎ ◎
Image Attributes ☓ ◯ ◯
Caption Generation ☓ ☓ ◯

28. Cloud vision API Limits
Type of Limit Usage Limit
MB per image 4 MB
MB per request 8 MB
Requests per second 10
Requests per feature per day 700,000
Requests per feature per month 20,000,000
Images per second 8
Images per request 16

29. Image recognition plugin
• embulk-filter-amazon_rekognition
• embulk-filter-azure_computer_vision_api
• embulk-filter-google_vision_api

30. Natural language processing
• Google Cloud Natural Language API
• Cognitive Services
– Language Understanding Intelligent Service
– Text Analytics API
– Bing Spell Check API

31. Natural language processing
AWS GCP Azure
Sentiment Analysis ☓ ◎ ◯
keyphrase Analysis ☓ ☓ ◎
Syntax Analysis ☓ ◎ ☓
Entity Analysis ☓ ◎ ☓
Topic Analysis ☓ ☓ ◎

32. NLP plugin
• embulk-filter-azure_text_analytics
• embulk-filter-google_natural_language_api

33. Machine translation
• Google Translate API
– Support neural machine translation(need
premium edition).
• Azure
– Support neural machine translation.
– Text and speech
• Translator Text API
• Translator Speech API

34. Machine translation
AWS GCP Azure
Normal ☓ ◎ ◎
NMT ☓ ◎ ◯

35. Machine translation plugin
• embulk-filter-azure_translator_api
• embulk-filter-google_translate_api

36. Embulk my requests
• Binary Type support.
– I want scripting image processing.
• Replace logger.
– I want to structure logs.
– I want use fluent logger.
• executor plugin more developper friendly.
– Pluggable resource control.

37. Thanks.