1) The document summarizes recent papers on deep learning techniques for text summarization including neural attention models for sentence summarization and abstractive text summarization using sequence-to-sequence RNNs. 2) It also discusses papers on transformer models using attention mechanisms, mixture of experts layers in neural networks, and effective approaches to attention-based neural machine translation. 3) Recent advances in deep learning like attention mechanisms, transformer models, and mixture of experts layers have led to improvements in natural language tasks like text summarization and machine translation.

1. 1
DEEP LEARNING JP
[DL Papers]
http://deeplearning.jp/
Generating Wikipedia by Summarizing Long Sequences
(ICLR 2018)
Toru Fujino, scalab, UTokyo

2. • . /0
• s ( -:: 8 2 I
• : p 2>2 > goo.gl/wSuuS9 k
• 1 2ILCG B iI rd i
e
• a Ird i I R l
• G ItW )
• B W DC noI2>> > : g

3. • , ) 1
• ,
•
• : ,
•
•
• :
•
•
• , , ( )
• , ,

4. 1
• .G DC L N
•
• 51 3 2 : 0 6 4 1 (
•
• R
• // 1 2 : /1 1:1 4 1 )
•
•
•
1) Rush et al. “A Neural Attention Model for Sentence Summarization”, EMNLP 2015
2) Nallapati et al. “Abstractive Text Summarization using Sequence-to-Sequence RNNs and Beyond”, CoNLL 2016

5. • (,
,, ),( )
• e /
• S
•
/
• 2 A / /
2) Nallapati et al. “Abstractive Text Summarization using Sequence-to-Sequence RNNs and Beyond”, CoNLL 2016
2)

6. • 2. 1
/
• 2

7. • 1 W R
a
•
• 2 G
• 00 . c
•
• 1 d
https://en.wikipedia.org/wiki/Deep_learning

8. • goo.gl/wSuuS9 ( )

9. • ,
1
2
3
4

10. •
•
• -
•

11. • 3 43 4 Y p
• CD 4 M i
• ac c 43 , 4 a
• d ac c n nY
r Ly
• ot ldA CN m m
3 43 e
• 4 , 3 43 m u
• ) 24 : 42 3 43 m
• s e
3) A. Vaswani et al. “Attention is All You Need”, NIPS 2017
4) N. Shazzer et al. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, ICLR 2017

12. • . .
• 3 .)
• .) A
• ( 3 .) : :
3) A. Vaswani et al. “Attention is All You Need”, NIPS 2017

13. • ( 2
• E !"
= [!%
"
, !'
"
, … , !)"
"
]
• E !+
= [!%
+
, !'
+
, … , !)+
+
]
• ) E
• A : D
5) M.-T. Luong et al. “Effective Approaches to Attention-based Neural Machine translation”, EMNLP 2015
5)

14. • ( - E
• : D
• : D
• ) : D
• E : A

15. •
• A , -
•

16. • K K V 2
5 /6 ) ,
• ( A 6
• 6

17. • L K 3 ASA
• V =
• .
• 11,/1 /
•

18. ) (
• )
•
/(
4) N. Shazzer et al. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”, ICLR 2017
4)

19. • 9 L
• 2 / 02 2 / 02 2 /
• 9
• M - - - 9 =
• -1 5

20. •

21. • - -
• -
• -
• :

22. •

24. )& (
• ( ()

26. •
•

27. • M
• / / -,/ p k lsr im e
• W lsr f :
• n s k
• a a > - - /
• - / y
• -2 2 - -, - / Wy
•
• ot C L c A
• L M d L C
• lsr : L