Machine Learning Platform in LINE Fukuoka
•
9 likes•60,495 views
The 1st Session of LINE Developer Meetup in Taiwan #6
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Machine Learning Platform in LINE Fukuoka
Similar to Machine Learning Platform in LINE Fukuoka (20)
More from LINE Corporation
More from LINE Corporation (20)
Recently uploaded
Recently uploaded (20)
Machine Learning Platform in LINE Fukuoka
- 2. Machine Learning Platform in LINE Fukuoka Kengo Tateishi, LFK Data Labs Team
- 3. LINE Fukuoka DataLabs. From Fukuoka, Japan. 立石 賢吾 TATEISHI KENGO @tkengo @tkengo
- 4. • Introduction to Data Labs • Machine learning lifecycle • Platform for machine learning Agenda
- 5. Introduction to Data Labs
- 6. • Support our various services from the view point “Data” • High level analysis for the services. • Machine learning model development. • Hadoop clusters as a Data Lake storing huge data. What is Data Labs?
- 8. Data Scientist ML Engineer • 2 roles in LFK Data Labs Team. What is Data Labs?
- 10. What is necessary for machine learning? Machine learning lifecycle
- 11. • Build a model to predict something. • Learn a new algorithm to improve accuracy. • Read papers to get more knowledge. • Understand what happens inside of the machine learning model. • And so on… What is necessary for machine learning? Machine learning lifecycle
- 12. • Build a model to predict something. • Learn a new algorithm to improve accuracy. • Read papers to get more knowledge. • Understand what happens inside of the machine learning model. • And so on… What is necessary for machine learning? ALL STEPS ARE IMPORTANT Machine learning lifecycle
- 13. • Build a model to predict something. • Learn a new algorithm to improve accuracy. • Read papers to get more knowledge. • Understand what happens inside of the machine learning model. • And so on… What is necessary for machine learning? However… Machine learning lifecycle
- 14. Machine learning lifecycle There are so many other things we have to do.
- 15. • Collect data used for model training. • Prepare a datastore to store the collected data. • Do pre-processing such as cleaning, normalization, completion, and so on. • Create a training environment (Distributed? Single?) • Expose our prediction model as an API. • Build an infrastructure to serve APIs. • Monitor accuracy online and offline regularly. • And others… Machine learning lifecycle There are so many other things we have to do.
- 16. Feedbac k Basic Flow Parameter Tuning Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Model Training Machine learning lifecycle
- 17. Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback • Everyone tends to learn these 2 parts. • But other parts are still important. Basic Flow Feedbac k Basic Flow Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Parameter Tuning Model Training Machine learning lifecycle
- 18. Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback • Everyone tends to learn these 2 parts. • But other parts are still important. • In addition… Model Training Parameter Tuning Basic Flow Prepare a storage Code review Test Prepare a dashboard CI Prepare servers Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback Basic Flow Feedbac k Basic Flow Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Parameter Tuning Model Training Machine learning lifecycle
- 19. Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback • Everyone tends to learn these 2 parts. • But other parts are still important. • In addition… Model Training Parameter Tuning Basic Flow • We need to iterate this process as a lifecycle continuously. Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback Model Training Parameter Tuning Basic Flow Prepare a storage Code review Test Prepare a dashboard CI Prepare servers Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Feedback Basic Flow Feedbac k Basic Flow Data Collection Data Storing Pre- Processing API Development Deployment Accuracy Monitoring Parameter Tuning Model Training Machine learning lifecycle
- 21. As you may know. To achieve rapid continuous development in machine learning, we need an environment for machine learning. Platform for machine learning
- 22. HDFS Sqoop Fluentd Flask App Hive PySpark Training Servers Our Platform Application side Code repository (Github) Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster
- 23. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Drone as a CI Importing HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Code repository (Github)
- 24. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Drone as a CI Storing HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Importing Code repository (Github)
- 25. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Drone as a CI Data understanding Experimental modeling HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Storing Importing Code repository (Github)
- 26. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Drone as a CI Developing something great… HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Data understanding Experimental modeling Storing Importing Code repository (Github)
- 27. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Drone as a CI Deployment Deployment Deployment HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Data understanding Experimental modeling Storing Importing Code repository (Github)
- 28. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Drone as a CI ETL Training HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster Deployment Deployment Deployment Data understanding Experimental modeling Storing Importing Code repository (Github)
- 29. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask App Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Drone as a CI Uploading Trained Model HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster ETL Training Deployment Deployment Deployment Data understanding Experimental modeling Storing Importing Code repository (Github)
- 30. HDFS Sqoop Fluentd Jupyter Notebook sklearn Tensorflow Flask Hive PySpark Kubernetes Cluster Training Servers Our Platform Application side Pre- Processing (ETL) Object Storage Drone as a CI Serving our model HDFS Sqoop Fluentd Flask Hive PySpark Training Servers Object Storage Drone as a CI Jupyter Notebook Pre- Processing (ETL) sklearn Tensorflow Kubernetes Cluster ETL Training Deployment Deployment Deployment Data understanding Experimental modeling Storing Importing Uploading Trained Model Code repository (Github) App
- 31. Summary
- 32. What is necessary for machine learning? Summary
- 33. • Of course, we need knowledge about training algorithms, parameter tuning, and so on. • In addition, the platform for machine learning is also important. • It makes our development very rapid, robust, and easy. What is necessary for machine learning? Summary
- 34. THANK YOU