The document discusses two papers presented at NeurIPS 2018 on AutoML:
1) "Massively Parallel Hyperparameter Tuning" which proposes an asynchronous successive halving algorithm to parallelize hyperparameter tuning. This approach improves on synchronous successive halving by avoiding mis-promoting configurations.
2) "Neural Architecture Optimization" which uses a neural network to learn embeddings of neural network architectures for optimization. The approach achieves state-of-the-art results on CIFAR-10 and shows good transferability to other tasks.
3. What is AutoML?
• Hyperparameter Optimization (HPO):
• Neural Architecture Search (NAS): NN
• Meta Learning:
3
“The user simply provides data,
and the AutoML system
automatically determines
approach that performs best for
particular applications.”
Futter et al., 2018
AutoML: Methods, Systems, Challenges
4. AutoML @ NeurIPS 2018
• AutoML
• HPO, NAS, Meta Learning
• Meta Learning
• AutoML Meetup @ Google AI
•
– System for ML
– Meta Learning
– NeurIPS 2018 Competition Track
4
5. Hyperparameter Optimization @ NeurIPS 2018
• Bayesian Optimization Meta-learning
• 10 @
– “Regret bounds for meta Bayesian optimization with an unknown
Gaussian process prior”
– “Automating Bayesian Optimization with Bayesian Optimization”
– etc.
• Systems for ML
– “Massively Parallel Hyperparameter Tuning”
– “Population Based Training as a Service”
– etc.
5
7. Meta Learning @ NeurIPS 2018
• Keywords: Model-agnostic Meta-Learning, Few-shot Learning,
Transfer Learning, etc.
• 20 @
– “Bayesian Model-Agnostic Meta Learning”
– “Meta-Reinforcement Learning of Structured Exploration Strategies”
– etc.
• Meta Learning
– HPO NAS
7
8. Competition Track: AutoML3 @ NeurIPS 2018
• AutoML3:
–
–
• Tree-parzen Estimator + LightGBM/XGBoost
8
Train&Test
Task A Task B Task C Task D Task E
28. 28
Chain-structured Space
Tree-structured Space
Multi-branch Network
Cell Search Space
…
Full Evaluation
Lower Fidelities
Learning Curve Extrapolation
One-shot Architecture Search
…
Reinforcement Learning
Evolutionary Search
Bayesian Optimization
Monte Carlo Tree Search
…