2. Linear Algebra
History:
The beginnings of matrices and determinants goes back to
the second century BC although traces can be seen back to
the fourth century BC. But, the ideas did not make it to
mainstream math until the late 16th
century.
The Babylonians around 300 BC studied problems which
lead to simultaneous linear equations.
The Chinese, between 200 BC and 100 BC, came much closer
to matrices than the Babylonians. Indeed, the text Nine
Chapters on the Mathematical Art written during the Han
Dynasty gives the first known example of matrix methods.
In Europe, two-by-two determinants were considered by
Cardano at the end of the 16th
century and larger ones by
Leibniz and, in Japan, by Seki about 100 years later.
3. What is a Matrix?
A matrix is a set of elements, organized into rows and columns
dc
ba
Rows
Columns
a and d are the diagonal elements.
b and c are the off-diagonal elements.
Matrices are like plain numbers in many ways: they can be
added, subtracted, and, in some cases, multiplied and
inverted (divided).
4. Examples:
[ ]δβα=
−
= b
d
b
A ;
1
1
Dimensions of a matrix:
Numbers of rows by numbers of columns. The Matrix A is
a 2x2 matrix, b is a 1x3 matrix.
A matrix with only one column or only one row is called a
vector.
If a matrix has an equal numbers of rows and columns, it is
called a square matrix. Matrix A, above, is a square matrix.
Usual Notation: Upper case letters => matrices
Lower case => vectors
5. Matrix multiplication
Multiplication of matrices requires a conformability condition
The conformability condition for multiplication is that the
column dimensions of the lead matrix A must be equal to the
row dimension of the lag matrix B.
If A is an (m x n) and B an (n x p) matrix (A has the same
number of columns a B has rows), then we define the product
of AB. AB is (m x p) matrix with its ij-th element is
What are the dimensions of the vector, matrix, and result?
[ ] [ ]131211
232221
131211
1211 cccc
bb
bbb
aaaB ==
=
[ ]231213112212121121121111 babababababa +++=
Dimensions: a(1 x 2), B(2 x 3) => c (1 x 3)
jk
n
j ijba∑ =1
6. Transpose Matrix
−
=′=>
−
=
49
08
13
401
983
AA:Example
The transpose of a matrix A is another matrix AT
(also written A
′) created by any one of the following equivalent actions:
- Write the rows (columns) of A as the columns (rows) of AT
- Reflect A by its main diagonal to obtain AT
Formally, the (i, j) element of AT
is the (j, i) element of A:
[AT
]ij
= [A]ji
If A is a m × n matrix => AT
is a n × m matrix.
(A')' = A
Conformability changes unless the matrix is square.
7. Special Matrices: Identity and Null
000
000
000
100
010
001
Identity Matrix is a square matrix with ones
along the diagonal and zeros everywhere else.
Similar to scalar “1.”
Null matrix is one in which all elements are
zero. Similar to scalar “0.”
Both are diagonal matrices => off-diagonal
elements are zero.
Both are symmetric and examples of idempotent
matrices. That is,
A = AT
and A = A2
= A3
= …
8. Basic Operations
Addition, Subtraction, Multiplication
++
++
=
+
hdgc
fbea
hg
fe
dc
ba
−−
−−
=
−
hdgc
fbea
hg
fe
dc
ba
++
++
=
dhcfdgce
bhafbgae
hg
fe
dc
ba
Just add elements
Just subtract elements
Multiply each row by
each column and add
=
kdkc
kbka
dc
ba
k Multiply each
element by the scalar
10. Laws of Matrix Addition & Multiplication
++
++
=
+
=+
22222121
12121111
2221
1211
2221
1211
abaa
abba
bb
bb
aa
aa
BA
Commutative law of Matrix Addition: A + B = B + A
++
++
=
+
=+
22222121
12121111
2221
1211
2221
1211
abab
abab
bb
aa
bb
bb
AB
Matrix Multiplication is distributive across Additions:
A (B+ C) = AB + AC (assuming comformability applies).
11. Matrix Multiplication
Matrix multiplication is generally not commutative. That is,
AB ≠ BA even if BA is conformable
(because diff. dot product of rows or col. of A&B)
−
=
=
76
10
,
43
21
BA
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
=
+−+
+−+
=
2524
1312
74136403
72116201
AB
( ) ( )( ) ( ) ( )
( ) ( ) ( ) ( )
−−
=
++
−+−+
=
4027
43
47263716
41203110
BA
12. Exceptions to non-commutative law:
AB=BA iff
B = a scalar,
B = identity matrix I, or
B = the inverse of A -i.e., A-1
Theorem: It is not true that AB = AC => B=C
Proof:
−−−=
−
=
−−
−=
132
111
212
;
011
010
111
;
321
101
121
CBA
Note: If AB = AC for all matrices A, then B=C.
13. Inverse of a Matrix
Identity matrix:
AI = A
=
100
010
001
3I
Notation: Ij is a jxj identity matrix.
Given A (mxn), the matrix B (nxm) is a right-inverse for A iff
AB = Im
Given A (mxn), the matrix C (mxn) is a left-inverse for A iff
CA = In
14. Theorem: If A (mxn), has both a right-inverse B and a left-inverse C,
thenC = B.
Proof:
We have AB=Im and CA=In.
Thus,
C(AB)= C Im = C and C(AB)=(CA)B= InB = B
=> C(nxm)=B(mxn)
Note:
- This matrix is unique. (Suppose there is another left-inverse D,
then D=B by the theorem, so D=C.)
- If A has both a right and a left inverse, it is a square matrix. It
is usually called invertible.
15. Inversion is tricky:
(ABC)-1
= C-1
B-1
A-1
Theorem: If A (mxn) and B (nxp) have inverses, then AB is
invertible and (AB)-1
= B-1
A-1
Proof:
We have AA-1
=In and A-1
A=Im
BB-1
=In and B-1
B=Ip
Thus,
B-1
A-1
(AB) = B-1
(A-1
A) B= B-1
InB = B-1
B = Ip
(AB) B-1
A-1
= A (BB-1
) A-1
= A In A-1
= A A-1
= Im
=> AB is invertible and (AB)-1
= B-1
A-1
More on this topic later
16. Transpose and Inverse Matrix
(A + B)' = A' + B'
If A' = A, then A is called a symmetric matrix.
Theorems:
- Given two conformable matrices A and B, then (AB)' = B'A'
- If A is invertible, then (A-1
)' = (A')-1
(and A' is also invertible).
16
17. Properties of Symmetric Matrices
Definition:
If A' = A, then A is called a symmetric matrix.
Theorems:
- If A and B are n x n symmetric matrices, then (AB)' = BA
- If A and B are n x n symmetric matrices, then (A+B)' = B+A
- If C is any n x n matrix, then B = C'C is symmetric.
Useful symmetric matrices:
V = X’X
P = X(X’X)-1
X’
M = I – P = I - X(X’X)-1
X’
18. 4.1 Application I: System of equations
Assume an economic model as system of linear equations with:
aij parameters, where i = 1.. n rows, j = 1.. m columns, and n=m
xi endogenous variables,
di exogenous variables and constants
nn
n
n
nm
m
m
nn d
d
d
x
x
x
ax
ax
ax
axa
axa
axa
2
1
2
22
12
211
22121
12111
=
=
=
+
+
+
+
+
+
19. A general form matrix of a system of linear equations
Ax = d where
A = matrix of parameters
x = column vector of endogenous variables
d = column vector of exogenous variables and constants
Solve for x*
dAx
d
d
d
x
x
x
aaa
aaa
aaa
nnnmnn
m
m
=
=
2
1
2
1
21
22221
11211
Questions: For what combinations of A and d there will zero,
one, many or an infinite number of solutions? How do we
compute (characterize) those sets of solutions?
Application: System of equations
20. Solution of a General-equation System
Theorem: Given A (mxn). If A has a right-inverse, then the
equation Ax = d has at least one solution for every d (mx1).
Proof:
Pick an arbitrary d. Let H be a right-inverse (so AH=Im).
Define x*=Hd.
Thus,
Ax* = A Hd = Imd = d => x* is a solution.
20
21. Theorem: Given A (mxn). If A has a left-inverse, then the
equation Ax=d has at most one solution for every d (mx1). That
is, if Ax=d has a solution x* for a particular d, then x* is unique.
Proof:
Suppose x* is a solution and z* is another solution. Thus, Ax*=d
and Az*=d. Let G be a left-inverse for A (so GA=In).
Ax*=d => GA x*= Gd
=> Inx* = x* = Gd.
Az*= d => GA z* = Gd
=> Inz* = z* = Gd.
Thus,
x*=z*=Gd. ■
22. Assume the 2x2 model
2x + y = 12
4x + 2y = 24
Find x*, y*:
y = 12 – 2x
4x + 2(12 – 2x) = 24
4x +24 – 4x = 24
0 = 0 ? indeterminate!
Why?
4x + 2y =24
2(2x + y) = 2(12)
One equation with two
unknowns
2x + y = 12
x, y
Conclusion:
not all simultaneous
equation models have
solutions
(not all matrices have inverses).
Problem with the previous proof? We’re assuming the left-
inverse exists (and there’s always a solution).
23. Linear dependence
A set of vectors is linearly dependent if any one of them can
be expressed as a linear combination of the remaining
vectors; otherwise, it is linearly independent.
Formal definition: Linear independence (LI)
The set {u1,...,uk} is called a linearly independent set of vectors
iff
c1 u1+....+ ckuk = θ => c1= c2=...=ck,=0.
Notes:
- Dependence prevents solving a system of equations. More
unknowns than independent equations.
- The number of linearly independent rows or columns in a
matrix is the rank of a matrix (rank(A)).
25. Upper and Lower Triangular Matrices
LT
021
012
000
UT
100
600
521
A square (n x n) matrix C is:
-Upper Triangular (UT) iff Cij=0 for i>j
(if the diagonal elements are all equal to 1,
we have a upper-unit triangular (UUT)matrix)
-Lower Triangular (LT) iff Cij=0 for i<j
(if the diagonal elements are all equal to 1,
we have a lower-unit triangular (LUT) matrix)
-Diagonal (D) iff Cij=0 for i ≠j
Theorem:
The product of the two UT (UUT) matrices is UT (UUT).
The product of the two LT (LUT) matrices is LT (LUT).
The product of the two D matrices is D.
26. Q: Why are we interested in these matrices?
Suppose Ax=d, where A is LT (with non-zero diagonal
terms). Then, the solutions are recursive (forward substitution).
Example:
x1 = d1
a21 x1 + a22 x2 = d2
a31 x1 +a32 x2 + a33 x3 = d3
Similarly, suppose Ax=d, where A is UT (with non-zero diagonal
terms). Then, the solutions are recursive (backward substitution).
Example:
a11 x1 +a12 x2 + a13 x3 = d1
a22 x2 + a23 x3 = d2
a31 x3 = d3
27. Finding a solution to Ax=d
Given A (n x n). Suppose we can decompose A into A=LU,
where L is LUT and U is UUT (with non-zero diagonal).
Then
Ax=d => LUx = d.
Suppose L is invertible=> Ux = L-1
d = c (or d = Lc)
=>Solve by forward substitution for c.
Then, Ux = c(Gaussian elimination) => solve by backward
substitution for x.
Theorem:
If A (n x n) can be decomposed A=LU, where L is LUT
and U is UUT (with non-zero diagonal), then Ax=d has a
unique solution for every d.
28. We can write a “symmetric” decomposition. Since U has non-
zero diagonal terms, we can write U=DU*, where U* is UUT.
Example:
==>
=
=
100
210
421
*;
500
030
002
;
500
630
842
UDU
Theorems:
If we can write A (n x n) as A=LDU, where L is LUT, D is
diagonal with non zero diagonal elements, and U is UUT, then L,
D, and U are unique.
If we can write A (n x n) as A=LDU, and A is symmetric,
then we can write A=LDL’.
29. Theorem: Cholesky decomposition
If A is symmetric, A=LDL’, and all diagonal elements of D are positive,
then A = HH’.
Proof:
Since A is symmetric, then A=LDL.
The product of a LUT matrix and a D matrix is a LUT
matrix.
Let D*=D1/2
and L be a LT matrix.
Then H=LD* is matrix is LT.
=> A=HH’. ■
The Cholesky decomposition is unique. It is used in the numerical
solution systems of equations, non-linear optimization, Kalman
filter algorithms, etc..
André-Louis Cholesky (1875–1918)
30. Cholesky decomposition: Application
30
System of Equations
If A is symmetric and A=LDL’, and all elements of D are
positive (A is a positive definite matrix), then we can solve
Ax = b by
(1) Compute the Cholesky decomposition A=HH’.
(2) Solve Hy = b for y,
(3) With y known, solve H’x = y for x.
Note: A-1
is never computed.
Ordinary Least squares
Systems of the form Ax = b with A symmetric and positive
definite are common in economics. For example, the normal
equations in ordinary –i.e., linear- least squares problems are of this
form. A-1
is never computed.
31. The EndThe End
Call us for more
Information:
www.iTutor.com
Visit
1-855-694-8886
