Notes on the low rank matrix approximation of kernel
1. NOTES ON THE LOW-RANK
MATRIX APPROXIMATION
OF KERNEL MATRICES
Hiroshi Tsukahara
Denso IT Laboratory, Inc.
Aug. 23 (Fri) 2013
2. KERNEL METHOD
Supervised Learning Problem
Solving in Reproducing Kernel Hilbert Spaces
( ){ }niYXyxD iin ,,2,1, =×∈= )(s.t.:Find ii xfyYXf =→
2
1 2
))(,(
1
min f
n
xfyl
n
ii
n
i
Ff
λ
+∑=
∈
( )∑=
=→
n
i
ii xfFX
1
s.t.:Assuming ϕαϕ
n
RinonOptimizati
(1)
ill-defined problem!!
cf. representer theorem
3. Kernel method
If the loss function is given by
the explicit form of is not necessary but their inner
products:
Define the mapping implicitly by a kernel function:
),(:)( ⋅= xkxϕ
( )2
)(
2
1
))(,( xfyxfyl −=
ϕ
),(:)(),( xxkxx ′=′ϕϕ
ϕ
RHS is called as a kernel function
4. Solution can formally be written as:
However, the complexity for computing the
solution is very high:
T
njiijini yyyyxxkyIK ),,,(and),(Kwhere])[( 21
1
==+= −
λα
)( 3
nO
5. LOW-RANK APPROXIMATION
Low-rank approximation of kernel matrices
Their rank is usually very low comparing to n.
Making use of this property, assume that the kernel
matrix can be written as
Then, the complexity of calculating the solution can
be reduced considerably, due to the formula:
( )[ ]T
r
T
nn
T
RIRRRIIRR
11 1
)(
−−
++=+ λ
λ
λ
nrrnRRRK T
<<×≈ hmatrix witiswhere
O(r2
n)
6. Rough sketch for the derivation of the formula
1
)( −
+ n
T
IRR λ
( )[ ].
1
,
1
,
1
,
1
,
1
1
1
2
32
T
r
T
n
T
T
rn
T
TT
rn
TTT
n
T
n
RIRRRI
R
RR
I
R
I
R
RRRR
I
R
I
RRRRRR
I
RR
I
−
−
+−=
+−=
+
+
−−=
+
−
+−=
+=
λ
λ
λλλ
λλλλ
λλλλ
λλ
7. There are several algorithms for deriving the
low-rank approximation:
Nystrom approximation
Incomplete Cholesky decompositon
