Preferred Infrastructure and Deep Learning Trends
•
78 likes•15,915 views
1. This document discusses the history and recent developments in natural language processing and deep learning. It covers seminal NLP papers from the 1990s through 2000s and the rise of neural network approaches for NLP from 2003 onward. 2. Recent years have seen increased research and investment in deep learning, with many large companies establishing AI labs in 2012-2014 to focus on neural network techniques. 3. The document outlines some popular deep learning architectures for NLP tasks, including neural language models, word2vec, sequence-to-sequence learning, and memory networks. It also introduces the Chainer deep learning framework for Python.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Preferred Infrastructure and Deep Learning Trends
Similar to Preferred Infrastructure and Deep Learning Trends (7)
More from Yuya Unno
More from Yuya Unno (20)
Recently uploaded
Recently uploaded (20)
Preferred Infrastructure and Deep Learning Trends
- 2. ! -2008 ! ! 2008-2011 t 1 ! t ! 2011- 1 t ! t ! ! Jubatus NLP 12014- t 12015,
- 3. NLP 1YANS ! YANS 19 ! ! -‐‑‒19/3 9/5 ! YANS 13 ! ! "
- 4. 1 ! !
- 7. 3 1. 2. t 3.
- 8. 1.
- 9. ! ! ---- ---- ! Web EC t Web ! 1CGM ---- ---- ! SNS1Twitter Facebook ! 1Line ! t
- 10. ! ! ! ! ! ! t WWW ! Blog ! SNS
- 12. ! EC t ! ! ! ! ! ! SNS ! ! t
- 13. ! EC t ! 11000 ! ! ! ! ! SNS ! !
- 16. ! ! ! ! ! ! 1 1 SNS ! t
- 17. ! ! ! ! ! ! 1
- 18. 2. t
- 20. ! !
- 21. t t : Siri https://www.apple.com/jp/ios/siri/ http://v-assist.yahoo.co.jp/ https://www.nttdocomo.co.jp/service/information/shabette_concier/
- 23. 11/2 ! ! 1970 t PC ! ! ! 2000 ! t
- 24. 12/2 ! ! RSS 2000 ! ! 1 ! 2000 !
- 25. ! ! t ! ! ! !
- 27. t
- 28. t ! !
- 29. ! ! <12:00 :25
- 30. t ! ! ! ! ! ! t
- 31. t ! t ! ! t ! t ! ! t
- 32. 3.
- 34. ! ! 1993: [Brown+93] ! 1996: [Berger+96] ! 2001: [Lafferty+01] ! t ! 2003: Latent Dirichlet Allocation [Blei+03] ! 2006: Pitman-Yor language model [Teh06] ! ! 2006: [Clarke+06][Riedel+06] ! 2010: [Koo+10][Rush+10] ! ! 2003: Neural language model [Bengio+03] ! 2010: Recurrent Neural Network [Mikolov+10] ! 2013: Skipgram Model (word2vec) [Mikolov+13]
- 35. ! ! 90 2010 !
- 36. ! !
- 37. 1. ! 2011: 30% # 20% ! 2012: 26% # 16% http://image-net.org/challenges/LSVRC/2012/ilsvrc2012.pdf
- 38. 2. ! 2012/3: Google Hinton DNNresearch ! 2012/4: Baidu Institute of Deep Learning ! 2012/8, 10: Yahoo! IQ Engines LookFlow ! 2012/12: Facebook AI Lab LeCun ! 2014/1: Google DeepMind ! 2014/5: Andrew Ng Baidu ! 2014/8: IBM SyNAPSE
- 39. 3. ! 2014/1 ! 2014/10 -‐‑‒ ! 2015/06
- 40. 4. t ! ! Sequence-to-sequence (2014, Google) ! ! Deep Q-Networks (2013, DeepMind) ! ! Variational Auto-Encoder (2014, UvA) ! Stochastic Backprop (2014, DeepMind) ! ! Neural Turing Machines (2014, DeepMind) ! ! Memory Networks (2014, Facebook) ! ! Show and Tell (2015, Google)
- 41. ! ! ! ! ! ! !
- 42. ! t ! ! ! 1 ! !
- 43. Neural Network Language Model (NNLM) [Bengio +03] ! N NN ! N-1
- 44. Recurrent Neural Network Language Model (RNNLM) [Mikolov+10] ! t-1 t ! NNLM N ! ! http://rnnlm.org
- 45. Skip-gram (word2vec) [Mikolov+13] ! ! 1CBOW ! Analogical reasoning ! 1word2vec
- 48. Show and Tell [Vinyals+15] ! ! CNN RNN CNN CNN RNN
- 51. ! libsvm, liblinear ! ! JUMAN, Chasen, MeCab ! ! Moses (GIZA++) ! ! Stanford CoreNLP ! ! word2vec ! Skipgram ! Theano, Caffe, Torch ! t !
- 52. Chainer v1.0.0 2015/06/09 Powerful Flexible Intuitive
- 53. Chainer ! Chainer ! Python h(f(x) + g(x)) x f g h+
- 56. t
- 57. PFI 2010 Xappy12012 ! 2012 Gunosy ! ! ! t
- 58. Xappy t ! ! Blog ! Twitter Facebook SNS ! ! 4G ! t ! ! etc. ! GUI t
- 59. ! ! ! ! t ! PC t ! ! ! ! 10 !
- 61. (1/5) ! [Brown+93] Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, Robert L. Mercer. The mathematics of statistical machine translation: parameter estimation. Computational Linguistics Vol. 19 (2), 1993. ! [Berger+96] Adam L. Berger, Vincent J. Della Pietra, Stephen A. Della Pietra. A Maximum Entropy Approach to Natural Language Processing. Computational Linguistics, Vol. 22 (1), 1996. ! [Lafferty+01] John Lafferty, Andrew McCallum, Fernando C. N. Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. ICML2001.
- 62. (2/5) ! [Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan. Latent Dirichlet Allocation. JMLR Vol. 3, 2003. ! [Teh06] Yee Whye Teh. A Hierarchical Bayesian Language Model based on Pitman-Yor Processes. ACL 2006. ! [Clarke+06] James Clarke, Mirella Lapata. Constraint-Based Sentence Compression: An Integer Programming Approach. COLING/ACL 2006. ! [Riedel+06] Sebastian Riedel, James Clarke. Incremental Integer Linear Programming for Non-projective Dependency Parsing. COLING/ACL 2006.
- 63. (3/5) ! [Koo+10] Terry Koo, Alexander M. Rush, Michael Collins, Tommi Jaakkola, David Sontag. Dual Decomposition for Parsing with Non-Projective Head Automata. EMNLP 2010. ! [Rush+10] Alexander M. Rush, David Sontag, Michael Collins, Tommi Jaakkola. On Dual Decomposition and Linear Programming Relaxations for Natural Language Processing. EMNLP 2010. ! [Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. A Neural Probabilistic Language Model. JMLR, 2003.
- 64. (4/5) ! [Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan "Honza" Cernocky, Sanjeev Khudanpur. Recurrent neural network based language model. Interspeech, 2010. ! [Mikolov+13] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. CoRR, 2013. ! [Socher+12] Richard Socher, Brody Huval, Christopher D. Manning, Andrew Y. Ng. Semantic Compositionality through Recursive Matrix-Vector Spaces. EMNLP2012. ! [Sutskever+14] I. Sutskever, O. Vinyals, Q. V. Le. Sequence to Sequence Learning with Neural Networks. NIPS 2014.
- 65. (5/5) ! [Vinyals+15] O. Vinyals, A. Toshev, S. Bengio, D. Erhan. Show and Tell: A Neural Image Caption Generator. arXiv:1411.4555, 2014. ! [Weston+15] J. Weston, S. Chopra, A. Bordes. Memory Networks. ICLR 2015 ! [Sukhbaatar+15] S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus. End-To-End Memory Networks. arXiv:1503.08895, 2015.