LEGaTO presentation at ACM Middleware 2020, by Isabelly Rocha, Nathaniel Morris, Lydia Y. Chen, Pascal Felber, Robert Birke and Valerio Schiavoni

1. 21st International Middleware Conference
December 7 - 11, 2020, Delft, Netherlands
PipeTune: Pipeline Parallelism of Hyper and System
Parameters Tuning for Deep Learning Clusters
Isabelly Rocha1, Nathaniel Morris2, Lydia Y. Chen3, Pascal Felber1, Robert Birke4, Valerio Schiavoni1
1University of Neuchâtel, 2The Ohio State University, 3TU Delft, 4ABB Research

2. Deep Learning
How many
neurons
should each
layer have?
How many
epochs
should it
run?
Which
learning rate
to define?
How many
layers to
use?

3. Hyperparameters Tuning
run
optimize()
Hyperparameters Model Parameters Score
n_layers = 3
n_neuros = 1024
learning_rate = 0.1
Weights
Optimization
60%
n_layers = 5
n_neuros = 512
learning_rate = 0.1
Weights
Optimization
75%

4. Hyperparameters Tuning
Hey, you going
to sleep?
Yes, now
shut up
What if you try 0.01 as
a learning rate?

5. Hyperparameters Autotuning
model dataset
hyper-
parameters
ranges
metric optimization
function
Hyperparameter Tuner
trained model
optimal hyper-
parameters
Google
Vizier
user

6. Auto-tuning: What is the problem?
Estimated Cost of
Tuning 6 Parameters
Cost[$]
0
22,5
45
67,5
90
EC2 Instances
m4.4xlarge m4.8xlarge m5.12xlarge m5.16xlarge m5.24xlarge
Tuning Time by
Number of Parameters
TuningTime
[hours]
0
1
2
3
4
Number of Parameters
1 2 3 4 5 6
The user can define only one objective function in the existing auto-tuning tools.
The chosen function is typically accuracy and the tuning performance is ignored.
Tuning duration grows exponentially with the number of parameters to be tuned.
Using more resources to improve the tuning performance is an expensive solution.

7. Auto-tuning: How to improve it?
1. Hyperparameters not only impact accuracy but also tuning duration and energy.
2. The optimal system parameters depend on the chosen hyperparameters.
Batch Size Impact
Difference[%]
-70
-60
-50
-40
-30
-20
-10
0
Batch Size
64 256 1024
Accuracy Duration Energy Cores Impact on Duration
DurationDifference[%]
-45
-30
-15
0
15
30
45
60
Number of Cores
2 4 8
Batch 64 Batch 256 Batch 1024
Baseline: batch size = 32. Baseline: number of cores = 1.

8. Hyperparameter Tuner

9. Hyper-parameter Tuner: Optimized

10. Evaluation: Setup
Baseline
Tune: Hyperparameter tuning only (i.e.,
no system parameter considered)
Workloads
Scenarios
Environment
I. Single Node (Intel E5-2620 with 8 cores)
Implemented on top of Keras and TensorFlow
II. Distributed Cluster (4x Intel E3-1275 with 8 cores)
Implemented on top of Spark using BigDL
I. Single-Tenancy
“Offline mode” showing results of running an
independent unseen HPT Job.
II. Multi-Tenancy
“Online mode” showing the averaged response
time of a synthetic trace with 90% load.

11. Evaluation Scenario I (Single-Node)
Model AccuracyAccuracy[%]
0
20
40
60
80
jacobi spkmeans bfs
Tune PipeTune
Tuning Duration
Time[s]
0
750
1500
2250
3000
jacobi spkmeans bfs
Tune PipeTune
Training Duration
Time[s]
0
10
20
30
40
jacobi spkmeans bfs
Tune PipeTune
Tuning Energy
Energy[kJ]
0
0,225
0,45
0,675
0,9
jacobi spkmeans bfs
Tune PipeTune

12. Evaluation Scenario II
Averaged Response Time
Time[s]
0
3000
6000
9000
12000
jacobi spkmeans bfs
Tune PipeTune
Single Node Distributed Cluster
Averaged Response Time
Time[s]
0
2375
4750
7125
9500
mnist new20 all
Tune PipeTune

13. Summary
• PipeTune is a novel approach for DNN tuning jobs;
• Leverages the combination of hyper with system parameter tuning to achieve high
model accuracy under low runtime and energy consumption;
• Experimental evaluation performed under various scenarios and using different state-
of-the-art workloads indicates promising results;
• Reduces the tuning time up to 23%;
• Speeds up the training time by up to 1.7x;
• Lowers energy consumption up to 20%;
• Refer to the paper for: more detailed evaluation and intermediate solution;
• Source code available in: https://github.com/isabellyrocha/pipetune.