Basics of RNNs and its applications with following papers: - Generating Sequences With Recurrent Neural Networks, 2013 - Show and Tell: A Neural Image Caption Generator, 2014 - Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, 2015 - DenseCap: Fully Convolutional Localization Networks for Dense Captioning, 2015 - Deep Tracking- Seeing Beyond Seeing Using Recurrent Neural Networks, 2016 - Robust Modeling and Prediction in Dynamic Environments Using Recurrent Flow Networks, 2016 - Social LSTM- Human Trajectory Prediction in Crowded Spaces, 2016 - DESIRE- Distant Future Prediction in Dynamic Scenes with Interacting Agents, 2017 - Predictive State Recurrent Neural Networks, 2017

1. Recurrent Neural
Network and its
Applications
in a nutshell
Sungjoon Choi
Kakao Brain

2. 2
Part 1: Basics of RNNs

3. Recurrent Neural Network
3
Basic structure
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

4. Recurrent Neural Network
4
Unfolding in time
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

5. Recurrent Neural Network
5
Long term dependency
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

6. Recurrent Neural Network
6
LongER term dependency
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

7. Long Short Term Memory
7
Vanilla RNN
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

8. Long Short Term Memory
8
Long Short Term Memory (LSTM)
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

9. Long Short Term Memory
9
Overall structure
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Input
Output
Cell state Next cell state
Hidden state Next hidden state
Forget gate
Input
gate
Output
gate

10. Long Short Term Memory
10
Core idea
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
vs.

11. Long Short Term Memory
11
Core idea
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

12. Long Short Term Memory
12
Step-by-step
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Forget gate
Input gate
Decide which information to throw away
from the cell state.
Decide which information to store to the
cell state.

13. Long Short Term Memory
13
Step-by-step
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Update
Output gate
Update the cell state scaled by input and
forget gates.
Output based on the updated cell state.

14. Long Short Term Memory
14
Reminder
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Input
Output
Cell state Next cell state
Hidden state Next hidden state
Forget gate
Input
gate
Output
gate

15. Gated Recurrent Unit
15
Variation of a LSTM
http://colah.github.io/posts/2015-08-Understanding-LSTMs/

16. Training a RNN
16
How do we train a RNN?

17. Training a RNN
17
How do we train a RNN?

18. Training a RNN
18
How do we train a RNN?
h1 = (WT
h0 + UT
x1)
h2 = (WT
(WT
h0 + UT
x1) + UT
x2)
h3 = (WT
(WT
(WT
h0 + UT
x1) + UT
x2) + UT
x3)
h3 = (WT
(WT
(WT
(WT
h0 + UT
x1) + UT
x2) + UT
x3) + UT
x4)

19. Training a RNN
19
How do we train a RNN?
h1 = (WT
h0 + UT
x1)
h2 = (WT
(WT
h0 + UT
x1) + UT
x2)
h3 = (WT
(WT
(WT
h0 + UT
x1) + UT
x2) + UT
x3)
h3 = (WT
(WT
(WT
(WT
h0 + UT
x1) + UT
x2) + UT
x3) + UT
x4)
Vanishing / exploding gradients often happen.

20. 20
Part 2: RNN Applications

21. Papers
21

22. 22
Cited over 1,000 times!

23. Prediction and Generation
23
How do we generate a sequence?

24. Prediction and Generation
24
How do we generate a sequence?
P(x) =
Y
t
P(xt|x1:t 1)

25. The Role of Memory
25
Need to remember the past to predict the future.

26. The Role of Memory
26
Need to remember the past to predict the future.

27. Basic Architecture
27

28. Prediction Network
28
The prediction network is optimized by minimizing the following loss:
TX
t=1
log P(xt|x1:t 1)
However, synthesis is NOT possible yet. Why?

29. Text Prediction
29
From Wikipedia

30. Text Prediction
30
From 구운몽

31. Handwriting Prediction
31
Mixture density network
The outputs of a neural network parametrize a mixture distribution.
The inputs consist of pen offsets and end of stroke.

32. Handwriting Prediction
32
Mixture density network
The input and output of a network
More specifically,

33. Handwriting Prediction
33
Mixture density network

34. Handwriting Synthesis
34
What is different?
Soft window
Characters

35. Handwriting Synthesis
35
Soft window handling
! ", $ %& "
'(
)
*(
+(
$

36. Biased Sampling
36

37. 37

38. Image Captioning
38
Overall architecture

39. Machine Translation Model
39

40. Language Model
40

41. Language Model
41

42. Language Model
42
<Start> cat
h0
y0

43. Language Model
43
<Start> cat sat
h0
y0
h1
y1

44. Language Model
44
<Start> cat sat on
h0
y0
h1
y1
h2
y2

45. Language Model
45
<Start> cat sat on mat
h0
y0
h1
y1
h2
y2
h3
y3

46. Training Phase
46

47. Training Phase
47

48. Training Phase
48

49. Training Phase
49

50. Training Phase
50

51. Test Phase
51

52. Test Phase
52

53. Test Phase
53

54. Test Phase
54

55. Test Phase
55

56. Test Phase
56

57. Test Phase
57

58. Test Phase
58

59. Results
59

60. Results
60

61. 61

62. Show, Attend, and Tell
62

63. Soft Attention Mechanism
63
CNN RNN

64. Soft Attention Mechanism
64
Attention

65. Soft Attention Mechanism
65

66. Results
66

67. Results
67

68. Results
68

69. Results
69

70. 70

71. DenseCap
71

72. Model Architecture
72
Object detection Image captioning
Soft spatial attention

73. Results
73

74. Results
74

75. 75

76. Deep Tracking
76
The goal is to reveal the occluded parts of a scene by learning to track objects
from raw sensor data.

77. Filtering with RNNs
77

78. Filtering with RNNs
78

79. Results
79

80. 80

81. Proposed Architecture
81
Binary
Occupancy
Grids
First Layer Output Second Layer Output

82. Proposed Architecture
82
Recurrent Flow Network

83. Update Rule
83

84. Update Rule
84
Context Propagation

85. Update Rule
85
Next Input

86. Update Rule
86
Correction
Context Propagation

87. Update Rule
87
Next Input
Propagation
Prediction

88. Update Rule
88

89. Results
89
Noise (5%)

90. Results
90
Noise (10%)

91. Results
91
Noise (20%)

92. Results
92
Noise (30%)

93. Results
93
Ford LiDar dataset

94. 94

95. Social LSTM
95
This paper proposed an LSTM model which can learn general human movement
and predict their future trajectories.
A new model called SocialLSTM which can jointly predict the paths of all the
people in a scene by taking into account the common sense rules and social
conventions is presented.

96. SocialLSTM
96

97. SocialLSTM
97
A new social pooling method is presented by incorporating a social hidden
tensor per each LSTM.
Social pooling

98. SocialLSTM
98
Pose estimation
A new social pooling method is presented by incorporating a social hidden
tensor per each LSTM.
Future position of a pedestrian is modeled via a bivariate Gaussian distribution:

99. Results
99

100. 100

101. DESIRE
101
This paper introduced a Deep Stochastic IOC RNN Encoder decoder framework,
DESIRE, for the task of future predictions of multiple interacting agents in
dynamic scenes.
Following mechanisms are focussed:
Diverse Sample Generation
IOC-based Ranking and Refinement
Scene Context Fusion

102. Proposed Architecture
102

103. Proposed Architecture
103
Sample Generation Module
Both previous and future trajectories are encoded with RNN Encoder1 and RNN
Encoder2.
Conditional VAE samples a latent vector and fed into RNN Decoder1 to generate
a future trajectory where the reconstruction error is minimized.

104. Results
104

105. 105

106. Predictive State Recurrent Neural Networks
106
Many of time series modeling methods can be categorized as either recursive
Bayes Filtering or Recurrent Neural Networks.
Recursive Bayes Filtering
Hidden Markov models (HMMs) or Kalman fitering (KF)
Predictive State Representation (PSR) is a variation on Bayes filters that
represents a state as the statistics of a distribution of features of future
observations.
Recurrent Neural Network (RNN)
RNNs model sequential data via a parameterized internal state and
update function.

107. Predictive State Recurrent Neural Networks
107
Proposed architecture

108. Results
108
Swimmer