HMM, MEMM, CRF メモ

P(X, Y ) = P(Y )P(X|Y ) =
Y
t
P(Yt|Yt 1)P(Xt|Yt)
Yt 1 Yt Yt+1
Xt+1XtXt 1
HMM
X Y
1
HMM
Viterbi
P(X, Y )

P(X) =
X
Y
P(X, Y )
arg max
Y
P(Y |X) = arg max
Y
P(X, Y )
P(X)
= arg max
Y
P(X, Y )
HMM
X
→
X Y
→ Viterbi
EM
→ Baum-Welch
※

t(x, s2) = P(x1, x2, Y2 = s2)
|S| = 3 4
sS
s1
s2
s3
sE
YS Y1 Y2 Y3 Y4 YE
X1 X2 X3 X4
s1
s2
s3
s1
s2
s3
s1
s2
s3
s2
x1 x2

3(x, s1) = P(x4, Y3 = s1)
|S| = 3 4
sS
s1
s2
s3
sE
YS Y1 Y2 Y3 Y4 YE
X1 X2 X3 X4
s1
s2
s3
s1
s2
s3
s1
s2
s3
x4
s1

ˆy = arg max
y
P(y|x)
Viterbi
t
1. sS
2. t = 1, ..., T – 1
3. sE
=
4. t = T – 1, ..., 1
Viterbi
←
x = x1 · · · xT x1 · · · xt si
O(|S|2
T)
t(x, si) = max
y1···yt 1
P(x1, · · · , xt, y1, · · · , yt 1, Yt = si)
1(x, si) = P(si|sS)P(x1|si)
t+1(x, si) = max
sj
[ t(x, sj)P(si|sj)] P(xt+1|si)
⇥t(x, si) = arg max
sj
[ t(x, sj)P(si|sj)]
max
y
P(x, y) = max
sj
[ T (x, sj)] P(sE|sj)
ˆyT = arg max
sj
T (x, sj)
ˆyt = t(x, ˆyt+1)
1

arg max
sj
4(x, sj) = s1
t(x, sj)
Viterbi
|S| = 3 4
1 2 3
s1 s1 s3 s1
s2 s1 s2 s3
s3 s2 s1 s1
sj
t
sS
s1
s2
s3
sE
YS Y1 Y2 Y3 Y4 YE
X1 X2 X3 X4
s1
s2
s3
s1
s2
s3
s1
s2
s3
x1 x2 x3 x4
sS s2 s3 s1 sEs1

✓ ¯✓ = (· · · , ¯i, · · · , ¯aij, · · · ,¯bik, · · · )
※O
✓ = ( 1, · · · , |S|, a11, · · · , a|S||S|, b11, · · · , b|S||O|)
t(si|✓) =
X
sj 2S
t(si, sj|✓) t = 1, · · · , T 1
¯⇥i = ¯P(si|sS, ✓) = 1(si|✓)
¯aij = ¯P(sj|si, ✓) =
PT 1
t=1 t(si, sj|✓)
PT 1
t=1 t(si|✓)
¯bik = ¯P(ok|si, ✓) =
P
t:xt=ok
t(si|✓)
PT
t=1 t(si|✓)

HMM
(features)
→
→
X Y
,
‘er’
HMM ‘er’
‘er’
, ‘er’ …

[[x = y]] =
(
1 if x = y
0 if x 6= y
MEMM
MEMM
X Y
※
( )
HMM
※
Yt 1 Yt Yt+1
Xt+1XtXt 1
Z(Xt, Yt 1)
Ps(Yt|Xt) =
1
Z(Xt, s)
exp
X
a
afa(Xt, Yt)
!
P(Y |X) =
Y
t
Ps(Yt|Xt)[[Yt 1 = s]]
s ME

f<begins-with-number,question>
= 1
features 2
Usenet FAQ
begins-with-number
begins-with-ordinal
begins-with-punctuation
begins-with-question-word
begins-with-subject
blank
contains-alphanum
contains-bracketed-number
contains-http
contains-non-space
contains-number
contains-pipe
contains-question-mark
contains-question-word
ends-with-question-mark
first-alpha-is-capitalized
indented
indented-1-to-4
indented-5-to-10
more-than-one-third-space
only-punctuation
prev-is-blank
prev-begins-with-ordinal
shorter-than-30
Xt 1
head, question,
answer, tail
t
question
f<b,s>(Xt, Yt) =
(
1 if b(Xt) is true and Yt = s
0 otherwise
※ 1
t question

(x(1)
, y(1)
), · · · , (x(n)
, y(n)
)
MEMM
Generalized Iterative Scaling
1. o, s C
※
2.
3. x
4.
5. 3, 4 s ME
fc(o, s) 0 8
o, sfc(o, s) = C
X
a
fa(o, s)
C =
X
a
fa(o, s)
˜E[fa] =
1
n
nX
i=1
1
m
(i)
s
X
t:yt 1=s
fa(x
(i)
t , y
(i)
t )
E[fa] =
1
n
nX
i=1
1
m
(i)
s
X
t:yt 1=s
X
y2S
Ps(y|xt, )fa(x
(i)
t , y)
new
a = a +
1
C
log
˜E[fa]
E[fa]
!

MEMM
s0
s5
s4
s6
s3
s1
s2
0.65
0.35
1
0.5
0.5 1
1
1 s0 → s1 → s2 → s3 : 0.325
x1 x2 x3 s0 s1 s2 s3 s0 s1 s4 s3
s0 s5 s6 s3
ME
x1 x2 x3
1
1
s0 → s1 → s4 → s3 : 0.325
s0 → s5 → s6 → s3 : 0.35

P(Y |X) =
1
Z(X)
exp
0
@
X
t,i
ifi(Yt 1, Yt, X, t) +
X
t,j
µjgj(Yt, X, t)
1
A
Z(X)
X
CRF
CRF
MEMM
HMM
Yt 1 Yt Yt+1

log
Y
t
P(Yt|Yt 1)P(Xt|Yt)
!
=
X
t
{log P(Yt|Yt 1) + log P(Xt|Yt)}
=
X
t
8
<
:
X
<s,s0>
[[Yt 1 = s0
]][[Yt = s]] log P(s|s0
) +
X
<o,s>
[[Xt = o]][[Yt = s]] log P(o|s)
9
=
;
=
X
<s,s0>
log P(s|s0
)[[Yt 1 = s0
]][[Yt = s]] +
X
t,<o,s>
log P(o|s)[[Xt = o]][[Yt = s]]
P(Y |X) =
1
P(X)
exp
0
@
X
t,<o,s>
<o,s>f<o,s>(Yt 1, Yt, X, t) +
X
t,<s,s0>
µ<s,s0>g<s,s0>(Yt, X, t)
1
A
X
o
exp( <o,s>) = 1
X
s
exp(µ<s,s0>) = 1
HMM CRF
CRF
P(X, Y ) = P(Y )P(X|Y ) =
Y
t
P(Yt|Yt 1)P(Xt|Xt)
<o,s> f<o,s>(Yt 1, Yt, X, t) g<s,s0>(Yt, X, t)µ<s,s0>

ˆyT = arg max
sm
T (x, sm)
x skt k
1. sS
2. k = 1, ..., T – 1
3.
4. t = T – 1, ..., 1
Viterbi
←
ˆyt = k(x, ˆyt+1)
1(x, sl) = h1(sS, sl, x)
k+1(x, sl) = max
sm
[ k(x, sm) + hk+1(sm, sl, x)]
⇥k(x, sl) = arg max
sm
[ k(x, sm) + hk+1(sm, sl, x)]
k(x, sl) = max
y1···yk 1
"k 1X
t=1
ht(yt 1, yt, x) + hk(yk 1, sl, x)
#
ht(Yt 1, Yt, X) =
X
i
X
j
µjgj(Yt, X, t)
ˆy = arg max
y
P(Y |X) = arg max
y
2
4
X
t,i
X
t,j
µjgj(Yt, X, t)
3
5

(|S| + 2) × (|S| + 2)
|S| + 2
|S| + 2
sS
sm
sE
sl
Mt(X)
Mt(sl, sm|X) = exp ht(sl, sm, X)
↵t(X)
0(Y |X) =
(
1 if Y = sS
0 otherwise
T +1(Y |X) =
(
1 if Y = sE
0 otherwise
t(X)T
= t 1(X)T
Mt(X)
t(X)
t(X) = Mt+1(X) t+1(X)
sS sEMt(X)
↵t(X)
t(X)

C MEMM
(x(1)
, y(1)
), · · · , (x(n)
, y(n)
)
Generalized Iterative Scaling
1. C
※
2.
3. x
4.
5. 3, 4
C =
X
t,i
fi(yt 1, yt, x, t) +
X
t,j
gj(yt, x, t)
c(x, y) = C
X
t,i
fi(yt 1, yt, x, t)
X
t,j
gj(yt, x, t)
new
i = i +
1
C
log
˜E[fi]
E[fi]
!
E[fi] =
1
n
nX
k=1
X
t
X
sl,sm
t 1(sl|x(k)
, , µ)Mt(sl, sm|x(k)
, , µ)⇥t(sm|x(k)
, , µ)
Z(x(k)| , µ)
fi(sl, sm, x(k)
, t)
Z(x) =
Y
t
Mt(x)
!
sS ,sE
P(Yt 1 = sl, Yt = sm|x, , µ)
˜E[fi] =
1
n
nX
k=1
X
t
fi(y
(k)
t 1, y
(k)
t , x(k)
, t)
c(x(k)
, y(k)
) 0 k = 1, · · · , n

• , ( ). . , 1999.
• A. McCallum, D. Freitag, and F. Pereira. Maximum entropy Markov models for
information extraction and segmentation. Proc. ICML, pp. 591-598, 2000.
• J. Lafferty, A. McCallum, and F. Pereira. Conditional random fields: probabilistic
models for segmenting and labeling sequence data. Proc. ICML, pp. 282-289
, 2001.
• Charles Elkan. Log-Linear Models and Conditional Random Fields. Notes for a
tutorial at CIKM, 2008.
• Hanna M. Wallach. Conditional Random Fields: An Introduction. Technical Report
MS-CIS-04-21. Department of Computer and Information Science, University of
Pennsylvania, 2004.
• , , . Conditional Random Fields
. , pp. 89-96, 2004.
• http://www.dbl.k.hosei.ac.jp/~miurat/readings/Nov0706b.pdf
• http://www.dbl.k.hosei.ac.jp/~miurat/readings/Nov0706a.pdf

Conditional Random Fields
X
Y1 Y2 Y3
Y4 Y5
Y5 Y4 Y6
X
cf. PRML8

p(y|x) =
1
Z
Y
C
C(yC|x) =
1
Z
exp
X
C
E(yC|x)
!
chain-structed CRFs
Y1 Y2
Y2 Y1 Y3
PRML8
X
Y1 Y2 Y3 Y4 Y5 Y6

i yi C
CRF
E(yC|x) =
X
j
jfj(yi 1, yi, x, i)
X
C
E(yC|x) =
X
C
X
j
jfj(yi 1, yi, x, i)
=
X
j
jFj(y, x)
Fj(y|x) =
X
C
jfj(yi 1, yi, x, i)
p(y|x) =
1
Z
exp
0
@
X
j
jFj(y, x)
1
A

HMM, MEMM, CRF メモ

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to HMM, MEMM, CRF メモ

Similar to HMM, MEMM, CRF メモ (20)

More from Takeshi Arabiki

More from Takeshi Arabiki (17)

Recently uploaded

Recently uploaded (20)

HMM, MEMM, CRF メモ