EchoBay is a framework that uses Bayesian optimization to efficiently optimize hyperparameters for echo state networks (ESN), a type of recurrent neural network. It reduces the expertise required for users and provides automatic hyperparameter selection. Bayesian optimization requires fewer data samples than grid search to find optimal hyperparameters. EchoBay allows non-experts to rapidly train ESN models through an easy-to-use interface without requiring code writing.

1. EchoBay
Machine Learning Made Easy
17-31/05/2019
NGC X, San Francisco
Giuseppe Franco g.franco4@studenti.unipi.it

2. 27 March 2019
06 March 2019
29 March 2019
11 March 2019

3. How can we Democratize
Machine Learning?
3
Democratic
Machine Learning

4. 1D Data
Text Analysis Audio Processing
Time Series
4
Time
Value

5. An accessible Machine Learning Tool should be:
Fast Easy to UseFlexible
Democratic
Machine Learning
5

6. Recurrent Neural
Network
x = Input
A = Active Unit
H = Output
6
The Input is processed by the Active Unit
The output returns back as input for next time steps

7. Long Short-Term
Memory Network
Vanilla RNN
Long Short Term Memory (LSTM) Networks
7

8. LSTM Limitations
8
Expertise Required High Amount of Data High Training Time

9. Echo State Network
9

10. ● Win
is Random, i.e. No Training
● W is Random, i.e. No Training
● Wout
is Trained, using Least Square
● Fewer Weights
● Less Data Required
● Efficient and Fast Training
Echo State Network
10

11. Echo State Network
Hyper-Params
● How large the Reservoir?
● How to scale Win?
● How to scale W?
● How to regularize the training?
● ...
The Hyper-Parameters have a fundamental role for
the practical application of ESN
11

12. Hyper-Parameters
Toy Example
Hyper-ParamOne
Hyper-Param Two
Performance
Bad
Good
Optimal
12

13. Hyper-ParamOne
Hyper-Param Two
Performance
Bad
Good
Optimal
Hyper-Parameters
Toy Example
12

14. Hyper-ParamOne
Hyper-Param Two
Performance
Bad
Good
Optimal
Hyper-Parameters
Toy Example
12

15. Hyper-ParamOne
Hyper-Param Two
Bad
Good
Optimal
Performance
Hyper-Parameters
Toy Example
12

16. Hyper-ParamOne
Hyper-Param Two
Bad
Good
Optimal
Performance
Hyper-Parameters
Toy Example
12
Test Every Possible Combination!
Naive Approach Time Consuming Possible Bias

17. Our Proposed Approach:
Bayesian Optimization
Hyper-ParamOne
Hyper-Param Two
Proposed Approach
Bad
Good
Optimal
Performance
13

18. Hyper-ParamOne
Hyper-Param Two
Proposed Approach
Smart Sampling
Unrelated to the
number of
Hyper-params
Fast Selection
Bad
Good
Optimal
Performance
Our Proposed Approach:
Bayesian Optimization
14

19. Bayesian Optimization reduces the required
ESN-related knowledge
Reducing Effort
User-Side
Modularity removes the necessity of code
writing
15

20. Our Solution:
EchoBay
EchoBay is a framework for a smart and efficient optimization of
Echo State Network.
16
User
Dataset
Configuration
File

21. Our Solution:
EchoBay
EchoBay is a framework for a smart and efficient optimization of
Echo State Network.
● Problem Definition
● Basic Hyper-Parameters Configuration
● Advanced Settings
16
User
Dataset
Configuration
File

22. EchoBay
Our Solution:
EchoBay
EchoBay is a framework for a smart and efficient optimization of
Echo State Network.
17
Automatic selection
of hyper-parameters
User
Dataset
Configuration
File

23. ● Create ESN Structures
● Train the Network
● Check performance on
Validation Set
● Upgrade Hyper-Param
Space
Our Solution:
EchoBay
EchoBay is a framework for a smart and efficient optimization of
Echo State Network.
17
User
Dataset
Configuration
File
Automatic selection
of hyper-parameters
EchoBay

24. User
Dataset
Configuration
File
Network Results &
Optimal Configuration
Structures for
replicating the
experiments
EchoBay
Automatic selection
of hyper-parameters
Our Solution:
EchoBay
EchoBay is a framework for a smart and efficient optimization of
Echo State Network.
18

25. Results
Grid Search vs Bayesian Search
19

26. Results
Grid Search vs Bayesian Search
19
Time 44 Minutes 13 Seconds
Error (NRMSE) 0.06 0.06
Performance

27. Conclusion
Rapid and smart setup and train
of ESN
Easy-to-use Machine Learning
framework
Reduces expertise required on
the user-side
20

28. Thanks for the attention!
Luca Cerina
Giuseppe Franco
Marco Santambrogio
Claudio Gallicchio
Alessio Micheli
luca.cerina@polimi.it
g.franco4@studenti.unipi.it
marco.santambrogio@polimi.it
gallicch@di.unipi.it
micheli@di.unipi.it
Rapid and smart setup and train
of ESN
Easy-to-use Machine Learning
framework
Reduces expertise required on
the user-side
Conclusion