104 DataFlow 從地上到雲端的無伺服器之路

1. DataFlow
從地上到雲端的無伺服器之路路
Senior Data Engineer
Gavin Lin
1

2. AGENDA
• Overview
• Why we upgrade
• What we want
• How we migrate
• Conclusions
2

3. Data
7.5
Millions
50
Millions
300
Billions
Resumes Jobs Behaviors
3Special thank Neil.Lee

4. Routine works
• Data processing
• Data mining (Marketing)
• Machine learning (Matching)
• Item-Item: Users who liked this
item also liked
• User-Item: Users who are similar
to you also liked
4Captain monkey - Sean.Chang

5. Why we upgrade
HDFS
Pig
v1.1.0
v0.11.0
• Resource utilization
• High-Availability
• Performance issue
5

6. What we want
• Resource Management. YARN
• High Availability. Zookeeper, Hadoop2
• Performance. Spark, Flink
• Streaming. Kafka
• Exploration. Kibana, Zeppelin, Jupyter
• Multi programming languages. Java, Python, Scala
6

7. Version II in IDC
HDFSv2
YARN
SparkFlink
Zookeeper
Kafka
Notebooks
HBase
Pig
ELK
7
Streaming Computing Exploration

8. Upgrade plan
Migration plan
8

9. Let’s go to AWS
Bottom-up &Top-down
9

10. How to migrate
ASF: 350+ Projects
AWS
10

11. An easy and expensive version
HDFS
YARN
SparkFlink
Zookeeper
Kafka
Notebook
HBase
Pig
ELK
11
Amazon
EMR
m5.large(0.124 USD) x 64台 x 24⼩小時 = 190 USD/day
128 cores, 512 G ram

12. “XXX as a Service” first
why not ?
12
Open Mind

13. Streaming
13
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

14. Full-Controllable or not
Amazon
Kinesis
Apache
Kafka
14
Unit Stream Topic
Distribution Shards Partitions
Thoughtput
2 MB read/shards
1 MB write/shards
Based on cluster size
Fault tolerance Handled by AWS Replica
Transformer AWS Lambda Connectors/Processors
Framework Support Spark and Flink

15. Storage
15
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

16. Cost / Performance
Amazon
S3
Amazon
EMR
Type Objects Block device
Throughput Middle High
Cost 0.025 USD/GB 0.12x3 USD/GB (EBS gp2)
Maintenance
No
(Policy / Lifecycle)
Yes
Libraries
Hadoop-aws
(3.x.x is better)
ALL
16
Storing Apache Hadoop Data on the Cloud - HDFS vs. S3
Top 5 Reasons for Choosing S3 over HDFS

17. Computing
17
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

18. Exploration
Online Analytical Processing
• Interactive interface for complexity and repeated
SQL query
18
Amazon
EMR
Warn man - ifegn.chen

19. ETL / ML
Orchestration
19
AWS Batch
https://kubernetes.io/blog/2018/06/28/airflow-on-kubernetes-part-1-a-different-kind-of-operator/

20. SUBMITTED
PENDING
Directed Acyclic Graph
20
RUNNABLE
STARTING
RUNNING
FAILED
SUCCEEDED
M, R family
Queue1
P family
Aggregation
One-hot
TrainingQueue2
One-hot
ETL
Submit Job
Dependencies
Job Definition 1
Job Definition 2
AWS Batch
Tool man - scott.hsieh
A typical machine learning workflow.

21. Artifact Docker image
JAR,Wheel
(Docker image)
Parallelism Independent Cluster
Scheduling YES No
Our scenario ETL / ML Exploration
Computing depends on
scenarios.
Amazon
EMR
AWS
Batch
21

22. UG Only - Something else
• ETL / ML
• DataPipeline,Athena, Glue, SageMaker,
• StepFunction with ECS
• Exploration
• Redshift, QuickSight
22

23. Event-Driven
23
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

24. Event eeee….
24
Amazon S3
alarm
event
(time-based)
event
(event-based)
AWS
CloudFormation
AWS Batch
Amazon
EMR
IAM
AWS
Config
Amazon Kinesis
Batch
Streaming
Auditting/Monitoring Processing
Notification
Justin - scott.liao

25. Serving(POC)
25
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
ServingETL, ML
Amazon
CloudWatch

26. API(POC)
Amazon ECS
26Boarder boy - chiyi.liao
K8S
Servie Management
Discovery
Monitoring
Registry
Logging
Authentication
Scheduling
Resource Management

27. AWS - DataFlow
Raw
Parquet
LogServer
AWS
Batch
Amazon
ECS
container
event
(time-based)
Amazon Kinesis
Firehose
Dataset/Model
Amazon
EMR
Exploration
Amazon
DynamoDB
Amazon
ElastiCache
container container
containercontainer container
containercontainer
containercontainer
Application
Load Balance
Serving
27
ETL, ML
Amazon
CloudWatch

28. Conclusions
28

29. • “XXX as a Service” first - why not
• Full-Controllable or not
• Cost / Performance (pay-as-you-go)
• Computing environment depends on scenario.
29

30. Speed UP
• Official Document
• Support
• Official
• Community
30

31. Thanks you all,
and my teammates.
31