Hyperparameter optimization is a common problem in machine learning. Machine learning algorithms, from logistic regression to neural nets, depend on well tuned hyperparameters to reach maximum effectiveness. Different hyperparameter optimization strategies have varied performance and cost (in time, money, and compute cycles.) So how do you choose? This talk will give a brief introduction to hyperparameter tuning and its importance, as well as the basics of how we apply statistical tests to make confident assertions about which hyperparameter optimization strategies can give you better results, faster.

1. Hyperparameter Optimization 101
Alexandra Johnson
Software Engineer, SigOpt

2. What are Hyperparameters?

3. Hyperparameters
affect model
performance

4. How Do I Find The Best Hyperparameters?

5. Step 1: Pick an Objective Metric
Classification models Accuracy
Regression models Root MSE

6. Caveat: Cross
Validate to Prevent
Overfitting

7. Cross Validation
4 5 60 1 2 3 7 8 9
4 5 6 7 8 90 1 2 3data
train validate metric

8. Cross Validation
4 5 60 1 2 3 7 8 9
4 5 6 7 8 90 1 2 3data
train
6 7 91 2 4 5 0 3 8train
7 8 90 2 3 6 1 4 5train
metric
metric
metric
Ktimes
validate
validate
validate

9. Grid Search Random Search Bayesian Optimization
Step 2: Pick an Optimization Strategy

10. Step 3: Evaluate N Times
N
Times

11. What is the Best Hyperparameter
Optimization Strategy?

12. Primary
Consideration: How
Good are the “Best”
Hyperparameters?

13. “Best Found Value” Distributions
experiments
accuracy

14. Secondary
Consideration: How
Much Time Do You
Have?

15. Number of Evaluations Required
Grid Search Random Search Bayesian
Optimization
2 parameters 100 ?? 20-40
3 parameters 1,000 ?? 30-60
4 parameters 10,000 ?? 40-80
5 parameters 100,000 ?? 50-100

16. SigOpt
Easy-to-use REST API,
R, Java, Python Clients
Ensemble of Bayesian
optimization techniques
Free trial, academic
discount, we’re hiring!

17. SigOpt Tutorial Videos
Versus untuned models:
+315.2% accuracy with TensorFlow CNN
+49.2% accuracy with Xgboost + unsupervised
features

18. Learn More
See more at sigopt.com/research:
● Blog posts
● Papers
● Videos

19. Thank You!
Twitter: @SigOpt
Email: support@sigopt.com
Web: sigopt.com/getstarted