Neil Tu LINE / Data Labs LINE's services are fast-growing and continuously generating different data and logs. In today's world, data scientists are put to the test to see how they are able to quickly extract valuable information – the modern asset – from massive data sets. Moreover, providing a stable, safe, and efficient mega-platform is imperative. This session will discuss the foundation that enables LINE's Data Labs team to continuously produce data.

1. Efficient and Invincible
Big Data Platform in LINE

2. Neil Tu (杜佐民)
● Data architect and engineer
● Expert on Hadoop distributed system and its ecosystems
● 5+ years of experience in image processing, computer vision,
and pattern recognition
About Me

3. Agenda • Data Platforms Within LINE
• Pipeline Platform
• Analysis Platform
• Ecosystem

4. Data Platforms Within LINE

5. Data Platforms

6. Data Platforms Big Data
Data Analysis
Mathematic Modeling
Pipeline
Machine Learning
Deep Learning
Etc.
Protocolized Model
System Integrated
Streaming

7. Tracking Service
Platform
Pipeline
Platform
Analysis
Platform
Data Platforms Within LINE

8. Pipeline Platform

9. Types
30
PB
6.5M
msg/sec
652

10. Service
System
ETL
Protocol definition
Data flow definition

11. Protocolized Data Model
message ApiAccessLog {
string request_id = 1;
string method = 2;
string path = 3;
string request_ip = 4 [(EsMapping.type) = "ip"];
int32 status = 5;
string contents = 6 [(EsMapping.index) = false];
string result = 7;
int64 event_time = 8 [(use_as_timestamp) = true];
int64 injest_time = 9;
}

12. Analysis Platform

13. Analysis Platform

14. Tables
25
PB
550
Users
1668

16. Data Infrastructure
BI tool
Event
log
RDBMS
dump
Other
storages
Data hub

17. Data Flow
RDBMS
ETL
Service data
Other
storages

18. Real-time Query
180,000 data / sec

19. ● UI
● Security
● Local backup
Nifi

20. Ecosystem

21. etc.
Oasis

22. https://github.com/yanagishima/yanagishima
Yanagishima

23. LINE Analytics
Reporting Tool

24. Interactive Data Analytics Tool
Oasis

25. Data Catalog Tool
Aquarium

26. Data Catalog Tool
Aquarium

27. Aquarium
Data Catalog Tool

28. Security

29. Office authentication Private authentication and authorization
Gateway server Client server

30. Sign-up web UI
HDFS
user home
directory
Registration
WF

31. Monitoring

32. ● JVM
● Net traffic
● Disk capacity
● etc.
Basic Monitoring

33. ● Small files
● Cluster usage per user
● Disk usage
● Blocks
● Empty files
● etc.
Cluster Monitoring

34. Third Namenode
NN1 NN2 NN3
JN1 JN2 JN3
Always on standby
Real-time metadata

35. Tuning

36. YARN
● yarn.log-aggregation.retain-check-interval-seconds=86400
● yarn.log-aggregation.retain-seconds=172800
Basic Tuning
Spark
● spark.history.fs.cleaner.enabled=true
● spark.history.fs.cleaner.interval=1d
● spark.history.fs.cleaner.maxAge=2d

37. Hive
● hive.merge.mapredfiles=true
● hive.merge.smallfiles.avgsize=128000000
● mapreduce.input.fileinputformat.split.maxsize=2147483648
● mapreduce.input.fileinputformat.split.minsize=134217728
● mapreduce.input.fileinputformat.split.minsize.per.node=134217728
● mapreduce.input.fileinputformat.split.minsize.per.rack=134217728
hive> ALTER TABLE xxx PARTITION (dt='19840312') CONCATENATE;
Basic Tuning

38. Conclusion

39. What is required?
● Be patient
How to achieve results?
● Trial and error
● Never give up
Running a Platform

40. THANK YOU