ETL & Machine Learning
•Download as PPTX, PDF•
4 likes•3,020 views
This slides was presented by me at PHPIndonesia and FemaleGeek Meetup on 18th June, 2016. On this occassion, I've shared about how Kudo start and organize our data team and more technically on how Kudo use and implement ETL and machine learning.
Report
Share
Report
Share
Recommended
Recommended
The machine learning libraries in Apache Spark are an impressive piece of software engineering, and are maturing rapidly. What advantages does Spark.ml offer over scikit-learn? At Data Science Retreat we've taken a real-world dataset and worked through the stages of building a predictive model -- exploration, data cleaning, feature engineering, and model fitting; which would you use in production?
The machine learning libraries in Apache Spark are an impressive piece of software engineering, and are maturing rapidly. What advantages does Spark.ml offer over scikit-learn?
At Data Science Retreat we've taken a real-world dataset and worked through the stages of building a predictive model -- exploration, data cleaning, feature engineering, and model fitting -- in several different frameworks. We'll show what it's like to work with native Spark.ml, and compare it to scikit-learn along several dimensions: ease of use, productivity, feature set, and performance.
In some ways Spark.ml is still rather immature, but it also conveys new superpowers to those who know how to use it.A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...
A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...Jose Quesada (hiring)
More Related Content
What's hot
The machine learning libraries in Apache Spark are an impressive piece of software engineering, and are maturing rapidly. What advantages does Spark.ml offer over scikit-learn? At Data Science Retreat we've taken a real-world dataset and worked through the stages of building a predictive model -- exploration, data cleaning, feature engineering, and model fitting; which would you use in production?
The machine learning libraries in Apache Spark are an impressive piece of software engineering, and are maturing rapidly. What advantages does Spark.ml offer over scikit-learn?
At Data Science Retreat we've taken a real-world dataset and worked through the stages of building a predictive model -- exploration, data cleaning, feature engineering, and model fitting -- in several different frameworks. We'll show what it's like to work with native Spark.ml, and compare it to scikit-learn along several dimensions: ease of use, productivity, feature set, and performance.
In some ways Spark.ml is still rather immature, but it also conveys new superpowers to those who know how to use it.A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...
A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...Jose Quesada (hiring)
What's hot (20)
The Quest for an Open Source Data Science Platform
The Quest for an Open Source Data Science Platform
Agile Machine Learning for Real-time Recommender Systems
Agile Machine Learning for Real-time Recommender Systems
Importance of ML Reproducibility & Applications with MLfLow
Importance of ML Reproducibility & Applications with MLfLow
Data Science Salon: Kaggle 1st Place in 30 minutes: Putting AutoML to Work wi...
Data Science Salon: Kaggle 1st Place in 30 minutes: Putting AutoML to Work wi...
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
ML-Ops: From Proof-of-Concept to Production Application
ML-Ops: From Proof-of-Concept to Production Application
A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...
A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...
Similar to ETL & Machine Learning
Similar to ETL & Machine Learning (20)
Kudo Codefest: Data science ETL & Predictive analytics to make better product
Kudo Codefest: Data science ETL & Predictive analytics to make better product
AI Deep Dive_ A Journey through Heroku_OpenAI Integration.pdf
AI Deep Dive_ A Journey through Heroku_OpenAI Integration.pdf
Building Data Science Pipelines in Python using Luigi
Building Data Science Pipelines in Python using Luigi
7 things Im excited about in the collaboration space
7 things Im excited about in the collaboration space
Oracle to PostgreSQL, Challenges to Opportunity.pdf
Oracle to PostgreSQL, Challenges to Opportunity.pdf
Recently uploaded
In the energy sector, the use of temporal data stands as a pivotal topic. At GRDF, we have developed several methods to effectively handle such data. This presentation will specifically delve into our approaches for anomaly detection and data imputation within time series, leveraging transformers and adversarial training techniques.Anomaly detection and data imputation within time series
Anomaly detection and data imputation within time seriesParis Women in Machine Learning and Data Science
(NEHA) Call Girls Katra Call Now: 8617697112 Katra Escorts Booking Contact Details WhatsApp Chat: +91-8617697112 Katra Escort Service includes providing maximum physical satisfaction to their clients as well as engaging conversation that keeps your time enjoyable and entertaining. Plus, they look fabulously elegant, making an impression. Independent Escorts Katra understands the value of confidentiality and discretion; they will go the extra mile to meet your needs. Simply contact them via text messaging or through their online profiles; they'd be more than delighted to accommodate any request or arrange a romantic date or fun-filled night together. We provide: (NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7Call Girls in Nagpur High Profile Call Girls
Model Call Girl Services in Delhi reach out to us at 🔝 9953056974 🔝✔️✔️
Our agency presents a selection of young, charming call girls available for bookings at Oyo Hotels. Experience high-class escort services at pocket-friendly rates, with our female escorts exuding both beauty and a delightful personality, ready to meet your desires. Whether it's Housewives, College girls, Russian girls, Muslim girls, or any other preference, we offer a diverse range of options to cater to your tastes.
We provide both in-call and out-call services for your convenience. Our in-call location in Delhi ensures cleanliness, hygiene, and 100% safety, while our out-call services offer doorstep delivery for added ease.
We value your time and money, hence we kindly request pic collectors, time-passers, and bargain hunters to refrain from contacting us.
Our services feature various packages at competitive rates:
One shot: ₹2000/in-call, ₹5000/out-call
Two shots with one girl: ₹3500/in-call, ₹6000/out-call
Body to body massage with sex: ₹3000/in-call
Full night for one person: ₹7000/in-call, ₹10000/out-call
Full night for more than 1 person: Contact us at 🔝 9953056974 🔝. for details
Operating 24/7, we serve various locations in Delhi, including Green Park, Lajpat Nagar, Saket, and Hauz Khas near metro stations.
For premium call girl services in Delhi 🔝 9953056974 🔝. Thank you for considering us!CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE9953056974 Low Rate Call Girls In Saket, Delhi NCR
Recently uploaded (20)
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Vip Mumbai Call Girls Marol Naka Call On 9920725232 With Body to body massage...
Anomaly detection and data imputation within time series
Anomaly detection and data imputation within time series
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
(NEHA) Call Girls Katra Call Now 8617697112 Katra Escorts 24x7
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Indiranagar Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call me @ 9892124323 Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
Call me @ 9892124323 Cheap Rate Call Girls in Vashi with Real Photo 100% Secure
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls in Sarai Kale Khan Delhi 💯 Call Us 🔝9205541914 🔝( Delhi) Escorts S...
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Saket (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
ETL & Machine Learning
- 1. ETL & Machine Learning @ Kudo FemaleGeek & PHP Indonesia Kudoplex 2, 18 Juni 2016
- 2. • Three years experienced software engineer in Java, Android and Python • Have big interest about Big Data and Data Science • Currently working as a Data Engineer at Kudo Luthfi Hariz Email : luthfi@kudo.co.id Linkedin : https://id.linkedin.com/in/luthfihariz
- 3. Data Analyst Predictive Analytics, Reporting Analysis, Fraud Analysis Data Engineer ETL, Data Infrastructure, Machine Learning at Scale
- 4. Why Kudo need Data team ? ● Our data is getting higher, especially in the variety ● Partnered with many vendor with different characteristic data ● Unique user (agent) behavior, not a typical e-commerce user ● Specific user (agent) profile
- 5. ETL (Extract Transform Load) Machine Learning
- 6. ETL (Extract Transform Load) We need ”analytics friendly” database that is single source of all data in Kudo
- 7. python package : petl, pandas Extract Transfor m Load operational & logs analytical DB
- 9. Tableau UI
- 10. ETL is all about jobs that run periodically we need to make sure all jobs run “pretty smooth..”
- 11. Airflow a platform to programmaticaly author, schedule and monitor our data pipelines
- 12. Support : • Retries • Complex Dependency (DAG) • Python Operator • Email on Error/Retry • Exchange Message between Task • Web UI • etc… Airflow
- 13. Airflow DAG (Directed Acyclic Graph) Task A Task B Task C Task D Task E Email OperatorPython Operator Retry 5 times HTTP Operator SQL Operator Bash Operator
- 14. Airflow Web UI
- 15. Airflow Web UI
- 16. Machine Learning
- 18. Category Product
- 19. Process “360 Degree Rotating Quiet Usb Fan” Elektronik 0.8 Fesyen 0.15 Perhiasan & Emas 0.05 Model Product Name Train Data 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 1 1 0 1 0 0 1 0 1 1 1 0 0 1 1 0 0 1 1 0 Naive Bayes
- 21. Recommendation Engine Item to Item Similiarity, Collaborative Filtering User Profilling Labelling Kudo Agents
- 22. Thank You!Luthfi Hariz - luthfi@kudo.co.id Pssst..we are hiring!