1. Matrix Factorization
Lecture #7
EEE 574
Dr. Dan Tylavsky

2. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Let’s look at solving Ax=b
A, b are dense.
Two possible (general approaches):
– Indirect Methods - methods which asymptotically approach (but never reach) the true solution as the number of steps increases.
Ex: Point Jacobi, Gauss-Siedel, Successive Over relaxation, Multigrid
–
•

3. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Direct Methods - methods that yield the theoretically exact result in a fixed number of steps. (Assuming an infinitely precise computing engine.)
Ex: Elimination Methods
Gauss Elimination
Gauss-Jordan Elimination
– Cholesky Factorization (Numerically Symmetric)
•
– LU or LDU factorization (a.k.a., Product form of the Inverse),(Variations Include Crout, Doolittle, Banachiewicz)
–
Orthogonalization Methods
–
QR factorization
–
Given Rotations
•
–
–

4. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Conjugate Gradient Methods - Exact solution in # of steps equal to the number of different eigenvalues.
Various Variations
Advantages of Elimination Methods
•
–
No convergence criteria (however, pivoting in general non-positive definite case.)
Factors can be used repeatedly.
Factors can be ‘easily’ modified to accommodate matrix/network changes.
–
Partial factorized matrices give linear network equivalents.
–
–
–

5. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Definitions
– Symmetric: A=AT
• Positive Definite: xTAx>0, (A>0 is short hand notation)
• Diagonally Dominant:
•
aii ≥ ∑a
j ≠i
,with inequality for at least one i.
ij
• Properly Diagonally Dominant: aii > ∑a
j ≠i
ij for all i.
• Rank: Order of the largest sub-matrix with nonzero determinant.

6. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Properties
– Symmetric pos. def. (A>0) matrix has all positive real eigenvalues.
• If A>0, then Det(A)=|A|>0.
• A rank 1 matrix must have the outer-product form, xyT.
•
1  2 4 6 
x =  2  , y T = [ 2 4 6]
  xy T = 4 8 12
 
 3
  6 12 18
 
• Symmetric A>0 has a unique Cholesky factorization. A=U U, A=U’ DU’
T T
• If A is properly diagonally dominant and all of its diagonal elements are positive, then A is positive definite.

7. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
– Factorizing a non-symmetric matrix.
 a11 a12 a13  a1n 
a a 22 a 23  a2n 
 21  α bT 
A =  a31 a32 a33  a3 n  =  1 ~ 
1
  c A1 
        1 
 a n1 an 2 a n3  ann 
 
– I claim A can be expanded in the following form:
α1 0 1 0 1  b1
T 
A=  α1  = L1 A 2U 1
 c1 I  0
 A2  0
 I  

8. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
– To find the value of A2, multiply.
α1 0 1 0 1  b1
T 
A=  α1  = L1 A 2U 1
 c1 I  0
 A2  0
 I  
 b1
T  α1 T
b1  
α1 0 1 α1 b1 
T
A=  α1  =  =
  c1 A1 
T
 c1b1 ~
 c1 A2  0 I   c1 A2 + 
  α1   
Outer product term of
– Therefore: first row & first column
T ~ T
~ c1b1 (absent. diag. element)
c1b1
∴ A2 + = A1 ⇒ A2 = A1 −
α1 α 1

9. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
– Do this repeatedly until we have the product form of A.
1 0T 
1 0  
T
T
U 1 = L1 0 α 2 b2 U 1
2
A = L1 A U 1 = L1 
0 A2 
  ~

 c2 A2 
1 0 0  1 0
T
0 T  1 0 0T  1 0 0T 
 T=
 
T  T
T
A 2 = 0 α 2 b2
b2  0 α 2 0  0 1 0  0 1 α2 
~  
 0 c2
 A2   0 c2 I  0 0 A3  0 0
  I 

= L2 A3 U2

10. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
– To find A3, multiply the result.
1 0 0T  1 0 0T  1 0 0T 
   T
b2 
A 2 = 0 α 2 T T
0  0 1 0  0 1 = L2 A3U 2
α2 
 0 c2
 I  0 0 A3  0 0
  I 

1 0 0T  1 0 0T  1 0 0T 
    
T
T
b2
 = 0 α 2
T
= 0 α 2 0  0 1 b2 
α2 
 0 c2 A3  0 0
 

I   0 c2 A3 + c2 b2
T 
 α2 

T ~ ~ c b T
c b
∴ A3 + 2 2 = A2 ⇒ A3 = A2 − 2 2
α2 α2
– We can write:
A = L1 L2 A3U 2U1

11. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
– Continuing on yields:
A = L1 L2  Ln −1 A nU n −1 U 2U 1
– where:
1 
  0 
An =  =L
n
 0 1 
 n 
 a nn 
– We can write:
A = L1 L2  Ln −1 LnU n −1 U 2U1

12. Elementary-Matrix-Multiplication-Superposition Principal.
–
© Copyright 1999 Daniel Tylavsky
Matrix Factorization
i j
1
i j

1  
    
  1 ←i
 αi  
Li L j =   αj  ← j ,i < j
 1  1 
 ci   
  cj  

 1 

 1

1 
  
 
 αi ←i
 
Li L j =  αj  ← j ,i < j
 1 
 
 ci c j  
 1
 

13. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
This implies I can write:
–
α1 
 α2 
 
 α3 
 
L1 L2  Ln −1 Ln =  c1  =L
 c2  
 
 c3  
 αn 
 

14. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Taking the transpose of the elementary-matrix- multiplication-superposition principal allows me to write:
–
 T
b1 
1 α1 
 T 
 1 b2 
 α2 
 T
b3 
U n −1U n − 2 U 2U 1 =  1
α3  = U
  
 
  
 1  
 

 1

Hence we can write:
–
A = LU

15. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Let’s work though an example:
–
10.000 -1.000 -2.000 -3.000
-1.000 12.100 -2.200 -2.700
A= -2.000 -2.200 15.880 -6.300
-3.000 -2.700 -6.300 23.400
Divide Top Row by a11.
–
T
~ c1b1 Divide 3rd Row by α3.
– Compute A2 = A1 −
–
α1
Divide 2nd Row by α2.
– T
~ c3b3
T
~ c2 b2
– Compute A4 = A3 −
α3
– Compute A3 = A2 −
α2
10.000 0 0 0
0
0.000 0.000 0.000 1.000
1.000 -0.100 -0.200 -0.300
-0.100 -0.200 -0.300
-0.100 -0.200 -0.300
-1.000 12.000 -2.400 -3.400
0 0
12.100 -2.200 -2.700
-3.000 00 1
11 0 0
-0.200 -0.250
-0.200 -0.250
0 0
A= -2.000 -2.400 15.480 -6.900 *
-2.200 15.880 -6.300 *
15.000 -7.500
0 00 0
00 11
1 0
-0.500
-0.500
0
-3.000 -3.000 -6.900 22.500
-2.700 -6.300 23.400
-7.500 21.750
18.000 00 0
00 00
0 11
1

16. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Individual Problem: Verify that previous answer is correct.
10.000 0 0 0 1.000 -0.100 -0.200 -0.300
-1.000 12.000 0 0 0 1 -0.200 -0.250
A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500
-3.000 -3.000 -7.500 18.000 0 0 0 1
Individual Problem: Factor A=LU for:
20.000 -20.000 -8.000 -8.000
-20.000 40.000 3.000 3.000
A= -8.000 3.000 14.450 -5.550
-8.000 3.000 -5.550 24.450
Note that result isn’t symmetrical.

17. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
To make this symmetric use the L’DL’T form.
– Let
l11  – Write:
 l 22 
  A = LD −1 DU = L' DL'T
D=
 l33 
 
 l 44 
10.000 0 0 0 1.000 -0.100 -0.200 -0.300
-1.000 12.000 0 0 0 1 -0.200 -0.250
A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500
-3.000 -3.000 -7.500 18.000 0 0 0 1
1 0 0 0 10 0 0 0 1.000 -0.100 -0.200 -0.300
-0.100 1 0 0 0 12 0 0 0 1 -0.200 -0.250
A= -0.200 -0.200 1 0 * 0 0 15 0 * 0 0 1 -0.500
-0.300 -0.250 -0.500 1 0 0 0 18 0 0 0 1
– Division by diag. of row & col. Performed @ same time.

18. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Individual Problem: Modify LU factors of A:
20.000 -20.000 -8.000 -8.000
-20.000 40.000 3.000 3.000
A= -8.000 3.000 14.450 -5.550
-8.000 3.000 -5.550 24.450
To get L’DL’T:

19. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Data structure for storing LU factors?
10.000 0 0 0 1.000 -0.100 -0.200 -0.300
-1.000 12.000 0 0 0 1 -0.200 -0.250
A= -2.000 -2.400 15.000 0 * 0 0 1 -0.500
-3.000 -3.000 -7.500 18.000 0 0 0 1
For LU factors could use LC-D-UR/X:
Pos: 0 1 2 3 4 5 6 7 8 9 10 11 12
DiagL(k) 10.0 12 15 18
U(k): -0.1 -0.2 -0.3 -0.2 -0.3 -0.5
L(k): -1.0 -2.0 -3.0 -2.4 -3.0 -7.5
Indx(k): 2 3 4 3 4 4
ERP(k): 0 3 5 6

20. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
1 0 0 0 10 0 0 0 1.000 -0.100 -0.200 -0.300
-0.100 1 0 0 0 12 0 0 0 1 -0.200 -0.250
A= -0.200 -0.200 1 0 * 0 0 15 0 * 0 0 1 -0.500
-0.300 -0.250 -0.500 1 0 0 0 18 0 0 0 1
For L’DL’T factors could use CR(L)-D/X:
Pos: 0 1 2 3 4 5 6 7 8 9 10 11 12
Diag(k) 10.0 12 15 18
L(k): -0.1 -0.2 -0.3 -0.2 -0.3 -0.5
RIndx(k): 2 3 4 3 4 4
ERP(k): 0 3 5 6

21. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
If we store & work through A by rows, factorization is done in a slightly different way:
–
10.000 -1.000 -2.000 -3.000
-1.000 12.100 -2.200 -2.700
A= -2.000 -2.200 15.880 -6.300
–
Divide Top Row by a11. -3.000 -2.700 -6.300 23.400
T
~ c1b1
– Compute A2 = A1 −
α1 for 2nd row.
Divide 2nd Row by α2.
–
T
~ c1b1 T
~ c2 b2
– Compute A2 = A1 −
α1and A3 = A2 − for
α2 3rd row.
10.000 0 0 0 1 -0.100 -0.200 -0.300
-1.000 12.000
12.100 0
-2.400
-2.200 0
-3.000
-2.700 0 1 -0.200
0 -0.250
0
A= -2.000 -2.400
-2.200 15.000
15.880 -7.500
-6.300 * 0 0 1 0
-3.000 -2.700 -6.300 23.400 0 0 0 1

22. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
Individual Problem: Finish the factorization.
10.000 0 0 0 1 -0.100 -0.200 -0.300
-1.000 12.000 0 0 0 1 -0.200 -0.250
A= -2.000 -2.400 15.000 -7.500 * 0 0 1 0
-3.000 -2.700 -6.300 23.400 0 0 0 1

23. © Copyright 1999 Daniel Tylavsky
Matrix Factorization
– Cholesky Factorization (Symmetric A=LLT.)
 a11 a12 a13  a1n 
a a 22 a 23  a2n 
 12  α bT 
A =  a13 a 23 a33  a 3n  =  1 ~ 
1
   b A1 
        1 
a1n a2n a3 n  a nn 
 
– A can be expanded in the following form:
 α1 0 1  T
b1 
0   α1
A =  b1 
α1  = L1 A L1
2 T
Etc.
 I  0 A2   

 α1 

 0
 I  
– Not used. Calculating square-roots is expensive.

24. The End