PyCon JP 2019 発表資料「PythonとAutoML」 データ分析の活用の幅の広がりに伴い、AutoMLの重要性が増してきました。本セッションでは、AutoMLの基礎事項から研究のトレンド、注目すべきPythonのOSSライブラリの紹介を行ないます。

1. Feature
Preprocessing
Raw Data
Feature
Selection
Feature
Model
Selection
Data Cleaning
Automated Machine Learning in Python
PyCon JP 2019
AI Lab
Python AutoML
Feature
Preprocessing
Raw Data
Feature
Selection
Feature
Model
Selection
Data Cleaning

2. CyberAgent AI Lab
Masashi SHIBATA
c-bata c_bata_
Python

3. Feature
Preprocessing
Feature
Selection
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning

4. Feature
Preprocessing
Feature
Selection
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning

5. 1
2
3
4
Automated Feature Engineering
AutoML
Automated Hyperparameter Optimization
Automated Algorithm(Model) Selection

6. Feature
Preprocessing
Feature
Selection
Feature
Construction
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning
Topic 1
AutoML

7.
Jeff Dean, “An Overview of Google's Work on AutoML and Future Directions” , ICML 2019
https://slideslive.com/38917182/an-overview-of-googles-work-on-automl-and-future-directions

8.
Automated Hyperparameter Optimization
Hyperopt, Optuna, SMAC3, scikit-optimize, …
Jeff Dean, “An Overview of Google's Work on AutoML and Future Directions” , ICML 2019
https://slideslive.com/38917182/an-overview-of-googles-work-on-automl-and-future-directions

9. Jeff Dean, “An Overview of Google's Work on AutoML and Future Directions” , ICML 2019
https://slideslive.com/38917182/an-overview-of-googles-work-on-automl-and-future-directions
HPO + Automated Feature Engineering
featuretools, tsfresh, boruta, …

10.
Automated Algorithm(Model) Selection
Auto-sklearn, TPOT, H2O, auto_ml, MLBox, …
Jeff Dean, “An Overview of Google's Work on AutoML and Future Directions” , ICML 2019
https://slideslive.com/38917182/an-overview-of-googles-work-on-automl-and-future-directions

11. Feature
Preprocessing
Feature
Selection
Feature
Construction
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning
Topic 2

12. Grid Search / Random Search

14. Eric Brochu, Vlad M. Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions,
with application to active user modeling and hierarchical reinforcement learning. 2010. arXiv:1012.2599.

15. Eric Brochu, Vlad M. Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions,
with application to active user modeling and hierarchical reinforcement learning. 2010. arXiv:1012.2599.
:
:

16. Eric Brochu, Vlad M. Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions,
with application to active user modeling and hierarchical reinforcement learning. 2010. arXiv:1012.2599.

17.
Eric Brochu, Vlad M. Cora, and Nando de Freitas. A tutorial on Bayesian optimization of expensive cost functions,
with application to active user modeling and hierarchical reinforcement learning. 2010. arXiv:1012.2599.

18.
Jamieson, K. G. and Talwalkar, A. S.: Non-stochastic Best Arm Identification
and Hyperparameter Optimization, in AIS-TATS (2016).
10 epochs
trial #1
trial #2
trial #3
trial #4
trial #5
trial #6
trial #7
trial #8
trial #9
30 epochs 90 epochs
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina,
Moritz Hardt, Benjamin Recht, and Ameet Talwalkar.
Massively parallel hyperparameter tuning. arXiv preprint arXiv:1810.05934, 2018.

23. TPE, Asynchronous Successive Halving, Median Stopping Rule
Define-by-Run
https://github.com/pfnet/optuna

25. rung 0
0.088 0.056 0.035 0.027
0.495
0.122
0.150
0.788
0.093
0.115
0.238
0.106
0.104 0.058
trial 0
trial 4
trial 2
trial 6
trial 24
trial 1
trial 8
trial 5
trial 18
trial 3
trial 7
rung 1 rung 2 rung 3
rung

26. 10 worker

27. scikit-optimize

28. Feature
Preprocessing
Feature
Selection
Feature
Construction
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning
Topic 3

29. 1. Feature Preprocessing Operators.
StandardScaler, RobustScaler,
MinMaxScaler, MaxAbsScaler,
RandomizedPCA, Binarizer, and
PolynomialFeatures.
2. Feature Selection Operators:
VarianceThreshold, SelectKBest,
SelectPercentile, SelectFwe, and
Recursive Feature Elimination (RFE).
AutoML feature preprocessing
M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter.
Efficient and robust automated machine learning.
In Neural Information Processing Systems (NIPS), 2015
R. S. Olson and J. H. Moore. Tpot: A tree-based pipeline optimization
tool for automating machine learning.
In Workshop on Automatic Machine Learning, 2016
TPOT Auto-sklearn

30. 1. Feature Preprocessing Operators.
StandardScaler, RobustScaler,
MinMaxScaler, MaxAbsScaler,
RandomizedPCA, Binarizer, and
PolynomialFeatures.
2. Feature Selection Operators:
VarianceThreshold, SelectKBest,
SelectPercentile, SelectFwe, and
Recursive Feature Elimination (RFE).
AutoML feature preprocessing
M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter.
Efficient and robust automated machine learning.
In Neural Information Processing Systems (NIPS), 2015
R. S. Olson and J. H. Moore. Tpot: A tree-based pipeline optimization
tool for automating machine learning.
In Workshop on Automatic Machine Learning, 2016
TPOT Auto-sklearn

31. TPOT Auto-sklearn AutoML
• featuretools: Deep feature synthesis
• tsfresh
Wrapper methods, Filter methods, Embedded methods
• scikit-learn Boruta

32. Feature
Preprocessing
Feature
Selection
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning

36.
63
794
Time Series FeatuRE extraction based on Scalable Hypothesis tests
https://github.com/blue-yonder/tsfresh

37. Feature
Preprocessing
Feature
Selection
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning

38. Guyon and A. Elisseeff. An introduction to variable and feature selection.
Journal of Machine Learning Research, 3:1157–1182, 2003.

39. Filter method
Wrapper method
sklearn.feature_selection.RFE(Recursive Feature Elimination), Boruta (boruta_py)
Embedded method
scikit-learn feature_importances_
Guyon and A. Elisseeff. An introduction to variable and feature selection.
Journal of Machine Learning Research, 3:1157–1182, 2003.

40. Feature
Preprocessing
Feature
Selection
Feature
Construction
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning
Topic 4
( )

41. https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html

42. • AutoML 2
• ML
• ML
•
AutoML as a CASH Problem
Combined Algorithm Selection and Hyperparameter optimization
M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter.
Efficient and robust automated machine learning. In Neural Information Processing Systems (NIPS), 2015

43. Using Optuna for CASH Problems
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py

44. Optuna for CASH Problem
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py
Algorithm Selection

45. Optuna for CASH Problem
def objective(trial):
iris = sklearn.datasets.load_iris()
x, y = iris.data, iris.target
classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
if classifier_name == 'SVC':
svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto')
else:
rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
classifier_obj = sklearn.ensemble.RandomForestClassifier(
max_depth=rf_max_depth, n_estimators=10)
score = sklearn.model_selection.cross_val_score(
classifier_obj, x, y, n_jobs=-1, cv=3)
accuracy = score.mean()
return accuracy
https://github.com/pfnet/optuna/blob/v0.16.0/examples/sklearn_simple.py
Hyperparameter optimization

46. • auto-sklearn
• TPOT
• h2o-3
• auto_ml (unmaintained)
• MLBox
Adithya Balaji, Alexander Allen Benchmarking Automatic Machine Learning Frameworks https://arxiv.org/pdf/1808.06492v1.pdf

47. AutoML
SMAC3
ChaLearn AutoML challenge 2 track
Auto-sklearn
import sklearn.metrics
import autosklearn.classification
X_train, X_test, y_train, y_test = train_test_split(…)
automl = autosklearn.classification.AutoSklearnClassifier(…)
automl.fit(X_train.copy(), y_train.copy(), dataset_name='breast_cancer')
print(automl.show_models())
predictions = automl.predict(X_test)
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions))
https://github.com/automl/auto-sklearn

48. TPOT
https://github.com/EpistasisLab/tpot
Tree-based Pipeline Optimization Tool for Automating Data Science

49.
TPOT
https://github.com/EpistasisLab/tpot
from tpot import TPOTClassifier
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(…)
tpot = TPOTClassifier(verbosity=2, max_time_mins=2)
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
tpot.export('tpot_iris_pipeline.py')

50. • 20 preprocessors
• 16 feature selectors,
• 1-hot encoding, missing
value imputation,
balancing, scaling
• 17 classifiers
• pre-defined
hyperparameter spaces
• 20 preprocessors
• 12 classifiers
• pre-defined
hyperparameter spaces
• Pipeline search space:
• Flexible: combining
tree-shaped pipelines
Auto-sklearn TPOT

51. Automated Neural
Architecture Search

52. Feature
Preprocessing
Feature
Selection
Model
Selection
Parameter
Optimization
Model
Validation
Data Cleaning

54. THANK YOU