1. 2014/02/06 PFI
Statistical Semantic
~
word2vec
Preferred Infrastructure
(@unnonouno)
~

2. (@unnonouno)
!
!
!
!
!
IBM
PFI

4. Semantics

5. [Bird+10]
10
10.1
10.2
10.3
10.4
10.5
10.6
10.7
10.8

6. [
+96]
5.
5.1
5.2
5.3
5.4

7. Wikipedia
!
!
!
!
!
!

8. !
!
!
Statistical Semantics
Statistical Semantics

9. Statistical Semantics Distributional Semantics
!
!
!

10. [Evert10]
NAACL2010 Stefan Evert
Semantic Models
Distributional

11. ???
[Evert10]

12. ???
2 cat
pig
knife
[Evert10]

13. dog
[Evert10]

14. (Distributional Hypothesis)
The Distributional Hypothesis is that words
that occur in the same contexts tend to have
similar meanings (Harris, 1954). (ACL wiki
)
!
!

15. (Statistical Semantics)
Statistical Semantics is the study of "how the
statistical patterns of human word usage can be
used to figure out what people mean, at least to
a level sufficient for information access” (ACL
wiki
)
!
!

17. [
13]
!
!
!

18. !
!
!
!
!
!
!
!

19. !
!
PFI
!
!
!
!
1

20. 3
!
!
ex:
!
etc…
ex:
!
-
etc…
!
!
!
ex:
NN
NN
etc…

21. : Latent Semantic Indexing (LSI),
Latent Semantic Analysis (LSA) [Deerwester+90]
!
!
!
!

22. LSI
k:
(SVD)
U
=
x
∑
x
i
i k
V

23. LSI
!
!
!
!
SVD

24. !
-
-
etc.
etc.
!
-
!
etc.

26. LSI
NMF
PLSI
LDA
NNLM
RNNLM
NTF
Skipgram
NN

27. !
LSI
!
Good
!
!
Bad
!

28. !
!
!

29. Probabilistic Latent Semantic
Indexing (PLSI) [Hofmann99]
!
LSI
!
!
!
ex:
LSI

30. PLSI
!
!
!
!
!
!
ex:

31. Latent Dirichlet Allocation (LDA) [Blei03]
PLSI
! PLSI
LDA
!

32. LDA
!
NLP
!
!
1

33. !
!
!
ex:
etc.
!
!
!
1.0

34. !
!
Good
!
Bad
!
!
LSI
SVD

35. Non-negative Matrix Factorization (NMF) [Lee
+99]
!
SVD
!
!
[Lee+99]

36. NMF = PLSI [Dinga+08]
!
NMF
PLSI
!
NMF
PLSI

37. Non-negative Tensor Factorization (NTF)
[Cruys10]
3
!
!
2
3

38. !
!
SVD

39. !
!
Good
!
Bad
!
!
word2vec

40. Neural Network Language Model (NNLM) [Bengio
+03]
!
!
N
NN
N-1

41. Recurrent Neural Network Language Model
(RNNLM) [Mikolov+10]
!
t-1
t
!
NNLM
N
!
!
http://rnnlm.org

42. RNNLM
!
[Mikolov+13a]

43. RNNLM
!
Transition-based parser
RNNLM
!
!
!
Stack recurrent
Transition-based parser

44. Skip-gram
(word2vec) [Mikolov+13b]
!
!
CBOW
!
Analogical reasoning
!
Parser

45. Skip-gram
[Mikolov+13b]
: w1, w2, …, wT
!
wi
c
vw
w
5

46. !

47. [Mikolov+13c]
!

48. word2vec
!
!
!
!
!
NMF

49. [Kim+13]
!
“good”
”best”
”better”

50. [Mikolov+13d]
!
!

51. NN
!
!
!
2013
!
!
!
Mikolov
15

52. !
N
!
!
!
NN
!
!
!
!
NN
N

53. !
NN
!
!
!
!
!
!

54. !
Statistical Semantics
!
3
!
!
!
NN
!
!
NN

55. 1
!
!
!
!
!
[Bird+10] Steven Bird, Ewan Klein, Edward Loper,
,
,
.
.
, 2010.
[
+96]
.
.
, 1996.
[Evert10] Stefan Evert.
Distributional Semantic Models. NAACL 2010 Tutorial.
[
13]
.
.
, 2013.
[Deerwester+90] Scott Deerwester, Susan T. Dumais, George W.
Furnas, Thomas K. Landauer, Richard Harshman.
Indexing by Latent Semantic Analysis. JASIS, 1990.

56. 2
!
!
!
!
!
[Hofmann99] Thomas Hofmann.
Probabilistic Latent Semantic Indexing. SIGIR, 1999.
[Blei+03] David M. Blei, Andrew Y. Ng, Michael I. Jordan.
Latent Dirichlet Allocation. JMLR, 2003.
[Lee+99] Daniel D. Lee, H. Sebastian Seung.
Learning the parts of objects by non-negative matrix factorization.
Nature, vol 401, 1999.
[Ding+08] Chris Ding, Tao Li, Wei Peng.
On the equivalence between Non-negative Matrix Factorization and
Probabilistic Latent Semantic Indexing. Computational Statistics &
Data Analysis, 52(8), 2008.
[Cruys10] Tim Van de Cruys.
A Non-negative Tensor Factorization Model for Selectional Preference
Induction. Natural Language Engineering, 16(4), 2010.

57. 3
!
!
!
!
NN 1
[Bengio+03] Yoshua Bengio, Réjean Ducharme, Pascal Vincent,
Christian Jauvin.
A Neural Probabilistic Language Model. JMLR, 2003.
[Mikolov+10] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan
"Honza" Cernocky, Sanjeev Khudanpur.
Recurrent neural network based language model.
Interspeech, 2010.
[Mikolov+13a] Tomas Mikolov, Wen-tau Yih, Geoffrey Zweig.
Linguistic Regularities in Continuous Space Word
Representations. HLT-NAACL, 2013.
[Mikolov+13b] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey
Dean.
Efficient Estimation of Word Representations in Vector Space.
CoRR, 2013.

58. 4
!
!
!
NN 2
[Mikolov+13c] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory
S. Corrado, Jeffrey Dean.
Distributed Representations of Words and Phrases and their
Compositionality. NIPS, 2013.
[Kim+13] Joo-Kyung Kim, Marie-Catherine de Marneffe.
Deriving adjectival scales from continuous space word
representations. EMNLP 2013.
,
[Mikolov+13d] Tomas Mikolov, Quoc V. Le, Ilya Sutskever.
Exploiting Similarities among Languages for Machine
Translation. CoRR, 2013.