Presentation slides on meta-learning methods, with an emphasis on methods that are applicable to various tasks(including reinforcement learning)

1. Model-Agnostic Meta-Learning for Fast
Adaptation of Deep Networks
Yoonho Lee
Department of Computer Science and Engineering
Pohang University of Science and Technology
September 14, 2017

2. Meta-learning
Learning to learn(faster, safer etc)

3. Meta-learning
RL methods take a long time to train: needs meta-learning
The meta-train set for humans would be: objects in real life,
experience playing different games etc

4. Previous Deep Meta-Learning Methods

5. Previous Deep Meta-Learning Methods
RNNs as learners[10][12][2][7]
Metric Learning[5][11]
Optimizer Learning[9][1]

6. Previous Deep Meta-Learning Methods
RNNs as learners

7. Previous Deep Meta-Learning Methods
RNNs as learners12
(assuming a sufficiently expressive RNN) Search space
includes all conceivable ML algorithms
Moves the burden of innovation to RNNs
Ignores advances achieved in ML by humans
Subpar results
1
Adam Santoro et al. “One-shot Learning with Memory-Augmented Neural
Networks”. In: ICML (2016).
2
Yan Duan et al. “RLˆ2: Fast Reinforcement Learning via Slow
Reinforcement Learning”. In: (2016).

8. Previous Deep Meta-Learning Methods
Metric Learning

9. Previous Deep Meta-Learning Methods
Metric Learning34
Learn a metric in input space
Specialized to one/few-shot classification(Omniglot,
MiniImageNet etc)
Cannot use in other problems (e.g. RL)
3
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. “Siamese Neural
Networks for One-shot Image Recognition”. In: ICML (2015).
4
Oriol Vinyals et al. “Matching Networks for One Shot Learning”. In:
NIPS (2016).

10. Previous Deep Meta-Learning Methods
Optimizer Learning

11. Previous Deep Meta-Learning Methods
Optimizer Learning56
Learn parameter update given gradients (search space includes
SGD, RMSProp, Adam etc)
Applicable to any architecture/task
Best performance on Omniglot, MiniImageNet
5
Sachin Ravi and Hugo Larochelle. “Optimization as a Model for Few-shot
Learning”. In: ICLR (2017).
6
Marcin Andrychowicz et al. “Learning to learn by gradient descent by
gradient descent”. In: NIPS (2016).

12. Model-Agnostic Meta-Learning for Fast Adaptation
of Deep Networks
Chelsea Finn Pieter Abbeel, Sergey Levine

13. ImageNet pre-training
Pretrain network on ImageNet classification, then fine-tune
network on new task.
Enables NNs to learn new vision tasks using relatively small
datasets.
This works because we have a huge labelled image dataset
and the manifold of images have a somewhat consistent
structure even between different datasets and tasks.
How do we bring ’Initialization as meta-learning’ to non-vision
domains such as speech/NLP/RL?

14. Model-Agnostic Meta-Learning

15. Model-Agnostic Meta-Learning

16. Model-Agnostic Meta-Learning
Supervised Learning

17. Model-Agnostic Meta-Learning
Few-shot Sine wave regression experiments

18. Model-Agnostic Meta-Learning
One/few-shot Classification experiments

19. Model-Agnostic Meta-Learning
Reinforcement Learning
Score function estimator in lines 6/10, since we cannot
backprop through environment dynamics

20. Model-Agnostic Meta-Learning
RL experiments
https://sites.google.com/view/maml

21. Model-Agnostic Meta-Learning
Extension7
7
Zhenguo Li et al. “Meta-SGD: Learning to Learn Quickly for Few Shot
Learning”. In: (2017).

22. Discussion
Parameter space noise (as opposed to policy space noise) has
been shown to result in more consistent exploration89. This
supports MAML’s idea.
Why does MAML not overfit when taking multiple gradient
steps?
Do we need to overwrite all weights during adaptation?
8
Matthias Plappert et al. “Parameter Space Noise for Exploration”. In:
(2017).
9
Meire Fortunato et al. “Noisy Networks for Exploration”. In: (2017).

23. References I
[1] Marcin Andrychowicz et al. “Learning to learn by gradient
descent by gradient descent”. In: NIPS (2016).
[2] Yan Duan et al. “RLˆ2: Fast Reinforcement Learning via
Slow Reinforcement Learning”. In: (2016).
[3] Chelsea Finn, Pieter Abbeel, and Sergey Levine.
“Model-Agnostic Meta-Learning for Fast Adaptation of Deep
Networks”. In: (2017).
[4] Meire Fortunato et al. “Noisy Networks for Exploration”. In:
(2017).
[5] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov.
“Siamese Neural Networks for One-shot Image Recognition”.
In: ICML (2015).
[6] Zhenguo Li et al. “Meta-SGD: Learning to Learn Quickly for
Few Shot Learning”. In: (2017).

24. References II
[7] Nikhil Mishra, Mostafa Rohaninejad, and Xi UC Chen Pieter
Abbeel Berkeley. “Meta-Learning with Temporal
Convolutions”. In: (2017).
[8] Matthias Plappert et al. “Parameter Space Noise for
Exploration”. In: (2017).
[9] Sachin Ravi and Hugo Larochelle. “Optimization as a Model
for Few-shot Learning”. In: ICLR (2017).
[10] Adam Santoro et al. “One-shot Learning with
Memory-Augmented Neural Networks”. In: ICML (2016).
[11] Oriol Vinyals et al. “Matching Networks for One Shot
Learning”. In: NIPS (2016).
[12] Jx Wang et al. “Learning to Reinforcement Learn”. In:
(2016).

25. Thank You