Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

01.04 orthonormal basis_eigen_vectors

60 views

Published on

A Introduction to Linear Algebra

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

01.04 orthonormal basis_eigen_vectors

  1. 1. Introduction to Math for Artificial Introduction Orthonormal Basis and Eigenvectors Andres Mendez-Vazquez March 23, 2020 1 / 127
  2. 2. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 2 / 127
  3. 3. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 3 / 127
  4. 4. The Dot Product Definition The dot product of two vectors v = [v1, v2, ..., vn]T and w = [w1, w2, ..., wn]T v · w = n i=1 viwi 4 / 127
  5. 5. Example!!! Splitting the Space? For example, assume the following vector w and constant w0 w = (−1, 2)T and w0 = 0 Hyperplane 5 / 127
  6. 6. Example!!! Splitting the Space? For example, assume the following vector w and constant w0 w = (−1, 2)T and w0 = 0 Hyperplane 5 / 127
  7. 7. Then, we have The following results g 1 2 = (−1, 2) 1 2 = −1 × 1 + 2 × 2 = 3 g 3 1 = (−1, 2) 3 1 = −1 × 3 + 2 × 1 = −1 YES!!! We have a positive side and a negative side!!! 6 / 127
  8. 8. This product is also know as the Inner Product Where An inner product · · · , · · · satisfies the following four properties (u, v, w vectors and α a escalar): 1 u + v, w = u, w + v, w 2 αv, w = α v, w 3 v, w = w, v 4 v, v ≥ 0 and equal to zero if v = 0. 7 / 127
  9. 9. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 8 / 127
  10. 10. The Norm as a dot product We can define the longitude of a vector v = √ v · v A nice way to think about the longitude of a vector 9 / 127
  11. 11. The Norm as a dot product We can define the longitude of a vector v = √ v · v A nice way to think about the longitude of a vector 9 / 127
  12. 12. Orthogonal Vectors We have that Two vectors are orthogonal when their dot product is zero: v · w = 0 or vT w = 0 Remark We want orthogonal bases and orthogonal sub-spaces. 10 / 127
  13. 13. Orthogonal Vectors We have that Two vectors are orthogonal when their dot product is zero: v · w = 0 or vT w = 0 Remark We want orthogonal bases and orthogonal sub-spaces. 10 / 127
  14. 14. Some stuff about Row and Null Space Something Notable Every row of A is perpendicular to every solution of Ax = 0 In a similar way Every column of A is perpendicular to every solution of AT x = 0 Meaning What are the implications for the Column and Row Space? 11 / 127
  15. 15. Some stuff about Row and Null Space Something Notable Every row of A is perpendicular to every solution of Ax = 0 In a similar way Every column of A is perpendicular to every solution of AT x = 0 Meaning What are the implications for the Column and Row Space? 11 / 127
  16. 16. Some stuff about Row and Null Space Something Notable Every row of A is perpendicular to every solution of Ax = 0 In a similar way Every column of A is perpendicular to every solution of AT x = 0 Meaning What are the implications for the Column and Row Space? 11 / 127
  17. 17. Implications We have that under Ax = b e = b − Ax Remember The error at the Least Squared Error. 12 / 127
  18. 18. Implications We have that under Ax = b e = b − Ax Remember The error at the Least Squared Error. 12 / 127
  19. 19. Orthogonal Spaces Definition Two sub-spaces V and W of a vector space are orthogonal if every vector v ∈ V is perpendicular to every vector w ∈ W. In mathematical notation vT w = 0 ∀v ∈ V and ∀w ∈ W 13 / 127
  20. 20. Orthogonal Spaces Definition Two sub-spaces V and W of a vector space are orthogonal if every vector v ∈ V is perpendicular to every vector w ∈ W. In mathematical notation vT w = 0 ∀v ∈ V and ∀w ∈ W 13 / 127
  21. 21. Examples At your Room The floor of your room (extended to infinity) is a subspace V . The line where two walls meet is a subspace W (one-dimensional). A more convoluted example Two walls look perpendicular but they are not orthogonal sub-spaces! Why? Any Idea? 14 / 127
  22. 22. Examples At your Room The floor of your room (extended to infinity) is a subspace V . The line where two walls meet is a subspace W (one-dimensional). A more convoluted example Two walls look perpendicular but they are not orthogonal sub-spaces! Why? Any Idea? 14 / 127
  23. 23. Examples At your Room The floor of your room (extended to infinity) is a subspace V . The line where two walls meet is a subspace W (one-dimensional). A more convoluted example Two walls look perpendicular but they are not orthogonal sub-spaces! Why? Any Idea? 14 / 127
  24. 24. For Example Something Notable 15 / 127
  25. 25. Yes!! The Line Shared by the Two Planes in R3 Therefore!!! 16 / 127
  26. 26. We have then Theorem The Null Space N (A) and the Row Space C AT , as the column space of AT , are orthogonal sub-spaces in Rn Proof First, we have Ax =       A1 A2 ... Am       x =       0 0 ... 0       Therefore Rows in A are perpendicular to x ⇒ Then x is also perpendicular to every combination of the rows. 17 / 127
  27. 27. We have then Theorem The Null Space N (A) and the Row Space C AT , as the column space of AT , are orthogonal sub-spaces in Rn Proof First, we have Ax =       A1 A2 ... Am       x =       0 0 ... 0       Therefore Rows in A are perpendicular to x ⇒ Then x is also perpendicular to every combination of the rows. 17 / 127
  28. 28. We have then Theorem The Null Space N (A) and the Row Space C AT , as the column space of AT , are orthogonal sub-spaces in Rn Proof First, we have Ax =       A1 A2 ... Am       x =       0 0 ... 0       Therefore Rows in A are perpendicular to x ⇒ Then x is also perpendicular to every combination of the rows. 17 / 127
  29. 29. Then Therefore The whole row space is orthogonal to the N (A) Better proof x ∈ N (A)- Hint What is AT y? x AT y = (Ax)T y = 0T y = 0 AT y are all the possible combinations of the row space!!! 18 / 127
  30. 30. Then Therefore The whole row space is orthogonal to the N (A) Better proof x ∈ N (A)- Hint What is AT y? x AT y = (Ax)T y = 0T y = 0 AT y are all the possible combinations of the row space!!! 18 / 127
  31. 31. Then Therefore The whole row space is orthogonal to the N (A) Better proof x ∈ N (A)- Hint What is AT y? x AT y = (Ax)T y = 0T y = 0 AT y are all the possible combinations of the row space!!! 18 / 127
  32. 32. A little Bit of Notation We use the following notation N (A) ⊥ C AT Definition The orthogonal complement of a subspace V contains every vector that is perpendicular to V . This orthogonal subspace is denoted by V ⊥ 19 / 127
  33. 33. A little Bit of Notation We use the following notation N (A) ⊥ C AT Definition The orthogonal complement of a subspace V contains every vector that is perpendicular to V . This orthogonal subspace is denoted by V ⊥ 19 / 127
  34. 34. A little Bit of Notation We use the following notation N (A) ⊥ C AT Definition The orthogonal complement of a subspace V contains every vector that is perpendicular to V . This orthogonal subspace is denoted by V ⊥ 19 / 127
  35. 35. Thus, we have Something Notable By this definition, the nullspace is the orthogonal complement of the row space. 20 / 127
  36. 36. Look at this The Orthogonality 21 / 127
  37. 37. Orthogonal Complements Definition The orthogonal complement of a subspace V contains every vector that is perpendicular to V . This orthogonal subspace is denoted by V ⊥, pronounced “V prep”. Something Notable By this definition, the nullspace is the orthogonal complement of the row space. After All Every x that is perpendicular to the rows satisfies Ax = 0. 22 / 127
  38. 38. Orthogonal Complements Definition The orthogonal complement of a subspace V contains every vector that is perpendicular to V . This orthogonal subspace is denoted by V ⊥, pronounced “V prep”. Something Notable By this definition, the nullspace is the orthogonal complement of the row space. After All Every x that is perpendicular to the rows satisfies Ax = 0. 22 / 127
  39. 39. Orthogonal Complements Definition The orthogonal complement of a subspace V contains every vector that is perpendicular to V . This orthogonal subspace is denoted by V ⊥, pronounced “V prep”. Something Notable By this definition, the nullspace is the orthogonal complement of the row space. After All Every x that is perpendicular to the rows satisfies Ax = 0. 22 / 127
  40. 40. Orthogonal Complements Definition The orthogonal complement of a subspace V contains every vector that is perpendicular to V . This orthogonal subspace is denoted by V ⊥, pronounced “V prep”. Something Notable By this definition, the nullspace is the orthogonal complement of the row space. After All Every x that is perpendicular to the rows satisfies Ax = 0. 22 / 127
  41. 41. Quite Interesting We have the following If v is orthogonal to the nullspace, it must be in the row space. Therefore, we can build a new matrix A = A v Problem The row space starts to grow and can break the law dim (R(A)) + dim (Ker (A)) = n. 23 / 127
  42. 42. Quite Interesting We have the following If v is orthogonal to the nullspace, it must be in the row space. Therefore, we can build a new matrix A = A v Problem The row space starts to grow and can break the law dim (R(A)) + dim (Ker (A)) = n. 23 / 127
  43. 43. Quite Interesting We have the following If v is orthogonal to the nullspace, it must be in the row space. Therefore, we can build a new matrix A = A v Problem The row space starts to grow and can break the law dim (R(A)) + dim (Ker (A)) = n. 23 / 127
  44. 44. Additionally The left nullspace and column space are orthogonal in Rm Basically, they are orthogonal complements. As always Their dimensions dim Ker AT and dim R AT add to the full dimension m. 24 / 127
  45. 45. Additionally The left nullspace and column space are orthogonal in Rm Basically, they are orthogonal complements. As always Their dimensions dim Ker AT and dim R AT add to the full dimension m. 24 / 127
  46. 46. We have Theorem The column space and row space both have dimension r. The nullspaces have dimensions n − r and m − r. Theorem The nullspace of A is the orthogonal complement of the row space C AT - Rn. Theorem The null space of AT is the orthogonal complement of the column space C (A) - Rm. 25 / 127
  47. 47. We have Theorem The column space and row space both have dimension r. The nullspaces have dimensions n − r and m − r. Theorem The nullspace of A is the orthogonal complement of the row space C AT - Rn. Theorem The null space of AT is the orthogonal complement of the column space C (A) - Rm. 25 / 127
  48. 48. We have Theorem The column space and row space both have dimension r. The nullspaces have dimensions n − r and m − r. Theorem The nullspace of A is the orthogonal complement of the row space C AT - Rn. Theorem The null space of AT is the orthogonal complement of the column space C (A) - Rm. 25 / 127
  49. 49. Splitting the Vectors The point of "complements" x can be split into a row space component xr and a nullspace component xn: x = xr + xn Therefore Ax = A [xr + xn] = Axr + Axn = Axr Basically Every vector goes to the column space. 26 / 127
  50. 50. Splitting the Vectors The point of "complements" x can be split into a row space component xr and a nullspace component xn: x = xr + xn Therefore Ax = A [xr + xn] = Axr + Axn = Axr Basically Every vector goes to the column space. 26 / 127
  51. 51. Splitting the Vectors The point of "complements" x can be split into a row space component xr and a nullspace component xn: x = xr + xn Therefore Ax = A [xr + xn] = Axr + Axn = Axr Basically Every vector goes to the column space. 26 / 127
  52. 52. Not only that Every vector b in the column space It comes from one and only one vector in the row space. Proof If Axr = Axr −→ the difference is in the nullspace xr − xr. It is also in the row space... Given that the nullspace and the row space are orthogonal. They only share the vector 0. 27 / 127
  53. 53. Not only that Every vector b in the column space It comes from one and only one vector in the row space. Proof If Axr = Axr −→ the difference is in the nullspace xr − xr. It is also in the row space... Given that the nullspace and the row space are orthogonal. They only share the vector 0. 27 / 127
  54. 54. Not only that Every vector b in the column space It comes from one and only one vector in the row space. Proof If Axr = Axr −→ the difference is in the nullspace xr − xr. It is also in the row space... Given that the nullspace and the row space are orthogonal. They only share the vector 0. 27 / 127
  55. 55. Not only that Every vector b in the column space It comes from one and only one vector in the row space. Proof If Axr = Axr −→ the difference is in the nullspace xr − xr. It is also in the row space... Given that the nullspace and the row space are orthogonal. They only share the vector 0. 27 / 127
  56. 56. Not only that Every vector b in the column space It comes from one and only one vector in the row space. Proof If Axr = Axr −→ the difference is in the nullspace xr − xr. It is also in the row space... Given that the nullspace and the row space are orthogonal. They only share the vector 0. 27 / 127
  57. 57. And From Here Something Notable There is a r × r invertible matrix there hiding inside A. If we throwaway the two nullspaces From the row space to the column space, A is invertible 28 / 127
  58. 58. And From Here Something Notable There is a r × r invertible matrix there hiding inside A. If we throwaway the two nullspaces From the row space to the column space, A is invertible 28 / 127
  59. 59. Example We have the matrix after echelon reduced A =    3 0 0 0 0 0 5 0 0 0 0 0 0 0 0    You have the following invertible matrix B = 3 0 0 5 29 / 127
  60. 60. Example We have the matrix after echelon reduced A =    3 0 0 0 0 0 5 0 0 0 0 0 0 0 0    You have the following invertible matrix B = 3 0 0 5 29 / 127
  61. 61. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 30 / 127
  62. 62. Assume that you are in R3 Something like 31 / 127
  63. 63. Simple but complex A simple question What are the projections of b = (2, 3, 4) onto the z axis and the xy plane? Can we use matrices to talk about these projections? First We must have a projection matrix P with the following property: P2 = P Why? Ideas? 32 / 127
  64. 64. Simple but complex A simple question What are the projections of b = (2, 3, 4) onto the z axis and the xy plane? Can we use matrices to talk about these projections? First We must have a projection matrix P with the following property: P2 = P Why? Ideas? 32 / 127
  65. 65. Simple but complex A simple question What are the projections of b = (2, 3, 4) onto the z axis and the xy plane? Can we use matrices to talk about these projections? First We must have a projection matrix P with the following property: P2 = P Why? Ideas? 32 / 127
  66. 66. Then, the Projection Pb First When b is projected onto a line, its projection p is the part of b along that line. Second When b is projected onto a plane, its projection p is the part of the plane. 33 / 127
  67. 67. Then, the Projection Pb First When b is projected onto a line, its projection p is the part of b along that line. Second When b is projected onto a plane, its projection p is the part of the plane. 33 / 127
  68. 68. In our case The Projection Matrices for the coordinate systems P1 =    0 0 0 0 0 0 0 0 1    , P2 =    0 0 0 0 1 0 0 0 0    , P3 =    1 0 0 0 0 0 0 0 0    34 / 127
  69. 69. Example We have the following vector b = (2, 3, 4)T Onto the z axis: P1b =    0 0 0 0 0 0 0 0 1       2 3 4    =    0 0 4    What about the plane xy Any idea? 35 / 127
  70. 70. Example We have the following vector b = (2, 3, 4)T Onto the z axis: P1b =    0 0 0 0 0 0 0 0 1       2 3 4    =    0 0 4    What about the plane xy Any idea? 35 / 127
  71. 71. We have something more complex Something Notable P4 =    1 0 0 0 1 0 0 0 0    Then P4b =    1 0 0 0 1 0 0 0 0       2 3 4    =    2 3 0    36 / 127
  72. 72. We have something more complex Something Notable P4 =    1 0 0 0 1 0 0 0 0    Then P4b =    1 0 0 0 1 0 0 0 0       2 3 4    =    2 3 0    36 / 127
  73. 73. Assume the following We have that a1, a2, ..., an in Rm. Assume they are linearly independent They span a subspace, we want projections into the subspace We want to project b into such subspace How do we do it? 37 / 127
  74. 74. Assume the following We have that a1, a2, ..., an in Rm. Assume they are linearly independent They span a subspace, we want projections into the subspace We want to project b into such subspace How do we do it? 37 / 127
  75. 75. Assume the following We have that a1, a2, ..., an in Rm. Assume they are linearly independent They span a subspace, we want projections into the subspace We want to project b into such subspace How do we do it? 37 / 127
  76. 76. This is the important part Problem Find the combination p = x1a1 + x2a2 + · · · + xnan closest to vector b. Something Notable With n = 1 (only one vector a1) this projection onto a line. This line is the column space of A Basically the columns are spanned by a single column. 38 / 127
  77. 77. This is the important part Problem Find the combination p = x1a1 + x2a2 + · · · + xnan closest to vector b. Something Notable With n = 1 (only one vector a1) this projection onto a line. This line is the column space of A Basically the columns are spanned by a single column. 38 / 127
  78. 78. This is the important part Problem Find the combination p = x1a1 + x2a2 + · · · + xnan closest to vector b. Something Notable With n = 1 (only one vector a1) this projection onto a line. This line is the column space of A Basically the columns are spanned by a single column. 38 / 127
  79. 79. In General The matrix has n columns a1, a2, ..., an The combinations in Rm are vectors Ax in the column space We are looking for the particular combination The nearest to the original b p = Ax 39 / 127
  80. 80. In General The matrix has n columns a1, a2, ..., an The combinations in Rm are vectors Ax in the column space We are looking for the particular combination The nearest to the original b p = Ax 39 / 127
  81. 81. First We look at the simplest case The projection into a line... 40 / 127
  82. 82. With a little of Geometry We have the following Projection 0 41 / 127
  83. 83. Therefore Using the fact that the projection is equal to p = xa Then, the error is equal to e = b − xa We have that a · e = 0 a · e = a · (b − xa) = a · b − xa · a = 0 42 / 127
  84. 84. Therefore Using the fact that the projection is equal to p = xa Then, the error is equal to e = b − xa We have that a · e = 0 a · e = a · (b − xa) = a · b − xa · a = 0 42 / 127
  85. 85. Therefore Using the fact that the projection is equal to p = xa Then, the error is equal to e = b − xa We have that a · e = 0 a · e = a · (b − xa) = a · b − xa · a = 0 42 / 127
  86. 86. Therefore We have that x = a · b a · a = aT b aT a Or something quite simple p = aT b aT a a 43 / 127
  87. 87. Therefore We have that x = a · b a · a = aT b aT a Or something quite simple p = aT b aT a a 43 / 127
  88. 88. By the Law of Cosines Something Notable a − b 2 = a 2 + b 2 − 2 a b cos Θ 44 / 127
  89. 89. We have The following product a · a − 2a · b + b · b = a 2 + b 2 − 2 a b cos Θ Then a · b = a b cos Θ 45 / 127
  90. 90. We have The following product a · a − 2a · b + b · b = a 2 + b 2 − 2 a b cos Θ Then a · b = a b cos Θ 45 / 127
  91. 91. With Length Using the Norm p = aT b aT a a = a b cos Θ a 2 a = b |cos Θ| 46 / 127
  92. 92. Example Project b =    1 1 1    onto a =    1 2 2    Find p = xa 47 / 127
  93. 93. Example Project b =    1 1 1    onto a =    1 2 2    Find p = xa 47 / 127
  94. 94. What about the Projection Matrix in general We have p = ax = aaT b aT a = Pb Then P = aaT aT a 48 / 127
  95. 95. What about the Projection Matrix in general We have p = ax = aaT b aT a = Pb Then P = aaT aT a 48 / 127
  96. 96. Example Find the projection matrix for b =    1 1 1    onto a =    1 2 2    49 / 127
  97. 97. What about the general case? We have that Find the combination p = x1a1 + x2a2 + · · · + xnan closest to vector b. Now you need a vector Find the vector x,find the projection p = Ax,find the matrix P. Again, the error is perpendicular to the space e = b − Ax 50 / 127
  98. 98. What about the general case? We have that Find the combination p = x1a1 + x2a2 + · · · + xnan closest to vector b. Now you need a vector Find the vector x,find the projection p = Ax,find the matrix P. Again, the error is perpendicular to the space e = b − Ax 50 / 127
  99. 99. What about the general case? We have that Find the combination p = x1a1 + x2a2 + · · · + xnan closest to vector b. Now you need a vector Find the vector x,find the projection p = Ax,find the matrix P. Again, the error is perpendicular to the space e = b − Ax 50 / 127
  100. 100. Therefore The error e = b − Ax aT 1 (b − Ax) = 0 ... ... aT n (b − Ax) = 0 Or    aT 1 ... aT n    [b − Ax] = 0 51 / 127
  101. 101. Therefore The error e = b − Ax aT 1 (b − Ax) = 0 ... ... aT n (b − Ax) = 0 Or    aT 1 ... aT n    [b − Ax] = 0 51 / 127
  102. 102. Therefore The Matrix with those rows is AT AT (b − Ax) = 0 Therefore AT b − AT Ax = 0 Or the most know form x = AT A −1 AT b 52 / 127
  103. 103. Therefore The Matrix with those rows is AT AT (b − Ax) = 0 Therefore AT b − AT Ax = 0 Or the most know form x = AT A −1 AT b 52 / 127
  104. 104. Therefore The Matrix with those rows is AT AT (b − Ax) = 0 Therefore AT b − AT Ax = 0 Or the most know form x = AT A −1 AT b 52 / 127
  105. 105. Therefore The Projection is p = Ax = A AT A −1 AT b Therefore P = A AT A −1 AT 53 / 127
  106. 106. Therefore The Projection is p = Ax = A AT A −1 AT b Therefore P = A AT A −1 AT 53 / 127
  107. 107. The key step was AT [b − Ax] = 0 Linear algebra gives this "normal equation" 1 Our subspace is the column space of A. 2 The error vector b − Ax is perpendicular to that column space. 3 Therefore b − Ax is in the nullspace of AT 54 / 127
  108. 108. When A has independent columns, AT A is invertible Theorem AT A is invertible if and only if Ahas linearly independent columns. 55 / 127
  109. 109. Proof Consider the following AT Ax = 0 Here, Ax is in the null space of AT Remember the column space and null space of AT are orthogonal complements. And Ax an element in the column space of A Ax = 0 56 / 127
  110. 110. Proof Consider the following AT Ax = 0 Here, Ax is in the null space of AT Remember the column space and null space of AT are orthogonal complements. And Ax an element in the column space of A Ax = 0 56 / 127
  111. 111. Proof Consider the following AT Ax = 0 Here, Ax is in the null space of AT Remember the column space and null space of AT are orthogonal complements. And Ax an element in the column space of A Ax = 0 56 / 127
  112. 112. Proof If A has linearly independent columns Ax = 0 =⇒ x = 0 Then, the null space Null AT A = {0} i.e AT A is full rank Then, AT A is invertible... 57 / 127
  113. 113. Proof If A has linearly independent columns Ax = 0 =⇒ x = 0 Then, the null space Null AT A = {0} i.e AT A is full rank Then, AT A is invertible... 57 / 127
  114. 114. Proof If A has linearly independent columns Ax = 0 =⇒ x = 0 Then, the null space Null AT A = {0} i.e AT A is full rank Then, AT A is invertible... 57 / 127
  115. 115. Finally Theorem When A has independent columns, AT A is square, symmetric and invertible. 58 / 127
  116. 116. Example Use Gauss-Jordan for finding if AT A is invertible A =    1 2 1 2 0 0    59 / 127
  117. 117. Example Given A =    1 0 1 1 1 2    and b =    6 0 0    Find x and p and P 60 / 127
  118. 118. Example Given A =    1 0 1 1 1 2    and b =    6 0 0    Find x and p and P 60 / 127
  119. 119. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 61 / 127
  120. 120. Now, we always like to make our life easier Something Notable Orthogonality makes easier to find x, p and P. For this, we will find the orthogonal vectors At the column space of A 62 / 127
  121. 121. Now, we always like to make our life easier Something Notable Orthogonality makes easier to find x, p and P. For this, we will find the orthogonal vectors At the column space of A 62 / 127
  122. 122. Orthonormal Vectors Definition The vectors q1, q2, ..., qn are orthonormal if qT i qj = 0 when i = j 1 when i = j Then A matrix with orthonormal columns is assigned the special letter Q Properties A matrix Q with orthonormal columns satisfies QT Q = I 63 / 127
  123. 123. Orthonormal Vectors Definition The vectors q1, q2, ..., qn are orthonormal if qT i qj = 0 when i = j 1 when i = j Then A matrix with orthonormal columns is assigned the special letter Q Properties A matrix Q with orthonormal columns satisfies QT Q = I 63 / 127
  124. 124. Orthonormal Vectors Definition The vectors q1, q2, ..., qn are orthonormal if qT i qj = 0 when i = j 1 when i = j Then A matrix with orthonormal columns is assigned the special letter Q Properties A matrix Q with orthonormal columns satisfies QT Q = I 63 / 127
  125. 125. Additionally Given that QT Q = I Therefore When Q is square, QT Q = I means that QT = Q−1: transpose = inverse. 64 / 127
  126. 126. Additionally Given that QT Q = I Therefore When Q is square, QT Q = I means that QT = Q−1: transpose = inverse. 64 / 127
  127. 127. Examples Rotation cos Θ − sin Θ sin Θ cos Θ Permutation Matrix 0 1 1 0 Reflection Setting Q = I − 2uuT with u a unit vector. 65 / 127
  128. 128. Examples Rotation cos Θ − sin Θ sin Θ cos Θ Permutation Matrix 0 1 1 0 Reflection Setting Q = I − 2uuT with u a unit vector. 65 / 127
  129. 129. Examples Rotation cos Θ − sin Θ sin Θ cos Θ Permutation Matrix 0 1 1 0 Reflection Setting Q = I − 2uuT with u a unit vector. 65 / 127
  130. 130. Finally If Q has orthonormal columns The lengths are unchanged How? Qx = xT QT Qx = √ xT x = x 66 / 127
  131. 131. Finally If Q has orthonormal columns The lengths are unchanged How? Qx = xT QT Qx = √ xT x = x 66 / 127
  132. 132. Remark Something Notable When the columns of A were a basis for the subspace. All Formulas involve AT A What happens when the basis vectors are orthonormal AT A simplifies to QT Q = I 67 / 127
  133. 133. Remark Something Notable When the columns of A were a basis for the subspace. All Formulas involve AT A What happens when the basis vectors are orthonormal AT A simplifies to QT Q = I 67 / 127
  134. 134. Remark Something Notable When the columns of A were a basis for the subspace. All Formulas involve AT A What happens when the basis vectors are orthonormal AT A simplifies to QT Q = I 67 / 127
  135. 135. Therefore, we have The following Ix = QT b and p = Qx and P = QIQT Not only that The solution of Qx = b is simply x = QT b 68 / 127
  136. 136. Therefore, we have The following Ix = QT b and p = Qx and P = QIQT Not only that The solution of Qx = b is simply x = QT b 68 / 127
  137. 137. Example Given the following matrix Verify that is a orthogonal matrix 1 3    −1 2 2 2 −1 2 2 2 −1    69 / 127
  138. 138. We have that Given that using orthonormal bases is good How do we generate such basis given an initial basis? Graham Schmidt Process We begin with three linear independent vectors a, b and c 70 / 127
  139. 139. We have that Given that using orthonormal bases is good How do we generate such basis given an initial basis? Graham Schmidt Process We begin with three linear independent vectors a, b and c 70 / 127
  140. 140. Then We can do the following Select a and rename it A Start with b and subtract its projection along a B = b − AT b AT A A Properties This vector B is what we have called the error vector e, perpendicular to a. 71 / 127
  141. 141. Then We can do the following Select a and rename it A Start with b and subtract its projection along a B = b − AT b AT A A Properties This vector B is what we have called the error vector e, perpendicular to a. 71 / 127
  142. 142. Then We can do the following Select a and rename it A Start with b and subtract its projection along a B = b − AT b AT A A Properties This vector B is what we have called the error vector e, perpendicular to a. 71 / 127
  143. 143. We can keep with such process Now we do the same for the new c C = c − AT c AT A A − BT c BT B B Normalize them To obtain the final result!!! q1 = A A , q1 = B B , q3 = C C 72 / 127
  144. 144. We can keep with such process Now we do the same for the new c C = c − AT c AT A A − BT c BT B B Normalize them To obtain the final result!!! q1 = A A , q1 = B B , q3 = C C 72 / 127
  145. 145. We can keep with such process Now we do the same for the new c C = c − AT c AT A A − BT c BT B B Normalize them To obtain the final result!!! q1 = A A , q1 = B B , q3 = C C 72 / 127
  146. 146. Example Suppose the independent non-orthogonal vectors a, b and c a =    1 −1 0    , c =    0 0 1    , d =    0 1 1    Then Do the procedure... 73 / 127
  147. 147. Example Suppose the independent non-orthogonal vectors a, b and c a =    1 −1 0    , c =    0 0 1    , d =    0 1 1    Then Do the procedure... 73 / 127
  148. 148. We have the following process We begin with a matrix A A = [a, b, c] We ended with the following matrix Q = [q1, q2, q3] How are these matrices related? There is a third matrix!!! A = QR 74 / 127
  149. 149. We have the following process We begin with a matrix A A = [a, b, c] We ended with the following matrix Q = [q1, q2, q3] How are these matrices related? There is a third matrix!!! A = QR 74 / 127
  150. 150. We have the following process We begin with a matrix A A = [a, b, c] We ended with the following matrix Q = [q1, q2, q3] How are these matrices related? There is a third matrix!!! A = QR 74 / 127
  151. 151. Notice the following Something Notable The vectors a and A and q1 are all along a single line. Then The vectors a, b and A, B and q1, q2 are all in the same plane. Further The vectors a, b, c and A, B, B and q1, q2, q2 are all in the same subspace. 75 / 127
  152. 152. Notice the following Something Notable The vectors a and A and q1 are all along a single line. Then The vectors a, b and A, B and q1, q2 are all in the same plane. Further The vectors a, b, c and A, B, B and q1, q2, q2 are all in the same subspace. 75 / 127
  153. 153. Notice the following Something Notable The vectors a and A and q1 are all along a single line. Then The vectors a, b and A, B and q1, q2 are all in the same plane. Further The vectors a, b, c and A, B, B and q1, q2, q2 are all in the same subspace. 75 / 127
  154. 154. Therefore It is possible to see that a1, a2, ..., ak They are combination of q1, q2, ..., qk [a1, a2, a3] = [q1, q2, q3]    qT 1 a qT 1 b qT 1 c 0 qT 2 b qT 2 c 0 0 qT 3 c    76 / 127
  155. 155. Therefore It is possible to see that a1, a2, ..., ak They are combination of q1, q2, ..., qk [a1, a2, a3] = [q1, q2, q3]    qT 1 a qT 1 b qT 1 c 0 qT 2 b qT 2 c 0 0 qT 3 c    76 / 127
  156. 156. Gram-Schmidt From linear independent vectors a1, a2, ..., an Gram-Schmidt constructs orthonormal vectors q1, q2, ..., qn that when used as column vectors in a matrix Q These matrices satisfy A = QR Properties Then R = QT A is a upper triangular matrix because later q s are orthogonal to earlier a s. 77 / 127
  157. 157. Gram-Schmidt From linear independent vectors a1, a2, ..., an Gram-Schmidt constructs orthonormal vectors q1, q2, ..., qn that when used as column vectors in a matrix Q These matrices satisfy A = QR Properties Then R = QT A is a upper triangular matrix because later q s are orthogonal to earlier a s. 77 / 127
  158. 158. Gram-Schmidt From linear independent vectors a1, a2, ..., an Gram-Schmidt constructs orthonormal vectors q1, q2, ..., qn that when used as column vectors in a matrix Q These matrices satisfy A = QR Properties Then R = QT A is a upper triangular matrix because later q s are orthogonal to earlier a s. 77 / 127
  159. 159. Therefore Any m × n matrix A with linear independent columns can be factored into QR The m × n matrix Q has orthonormal columns. The square matrix R is upper triangular with positive diagonal. We must not forget why this is useful for least squares AT A = RT QT QR = RT R Least Squared Simplify to RT Rx = RT QT b or Rx = QT b or x = R−1QT b 78 / 127
  160. 160. Therefore Any m × n matrix A with linear independent columns can be factored into QR The m × n matrix Q has orthonormal columns. The square matrix R is upper triangular with positive diagonal. We must not forget why this is useful for least squares AT A = RT QT QR = RT R Least Squared Simplify to RT Rx = RT QT b or Rx = QT b or x = R−1QT b 78 / 127
  161. 161. Therefore Any m × n matrix A with linear independent columns can be factored into QR The m × n matrix Q has orthonormal columns. The square matrix R is upper triangular with positive diagonal. We must not forget why this is useful for least squares AT A = RT QT QR = RT R Least Squared Simplify to RT Rx = RT QT b or Rx = QT b or x = R−1QT b 78 / 127
  162. 162. Algorithm Basic Gram-Schmidt 1 for j = 1 to n 2 v = A (:, j) 3 for i = 1 to j − 1 4 R (i, j) = Q (:, i)T v 5 v = v − R (i, j) Q (:, i) 6 R (j, j) = v 7 Q (:, j) = v R(j,j) 79 / 127
  163. 163. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 80 / 127
  164. 164. A as a change factor Most vectors change direction when multiplied against a random A Av −→ v Example 81 / 127
  165. 165. A as a change factor Most vectors change direction when multiplied against a random A Av −→ v Example 81 / 127
  166. 166. However There is a set of special vectors called eigenvectors Av = λv Here, the eigenvalue is λ and the eigenvector is v. Definition If T is a linear transformation from a vector space V over a field F, T : V −→ V , then v = 0 is an eigenvector of T if T (v) is a scalar multiple of v. Something quite interesting Such linear transformations can be expressed by matrices A, T (v) = Av 82 / 127
  167. 167. However There is a set of special vectors called eigenvectors Av = λv Here, the eigenvalue is λ and the eigenvector is v. Definition If T is a linear transformation from a vector space V over a field F, T : V −→ V , then v = 0 is an eigenvector of T if T (v) is a scalar multiple of v. Something quite interesting Such linear transformations can be expressed by matrices A, T (v) = Av 82 / 127
  168. 168. However There is a set of special vectors called eigenvectors Av = λv Here, the eigenvalue is λ and the eigenvector is v. Definition If T is a linear transformation from a vector space V over a field F, T : V −→ V , then v = 0 is an eigenvector of T if T (v) is a scalar multiple of v. Something quite interesting Such linear transformations can be expressed by matrices A, T (v) = Av 82 / 127
  169. 169. A little bit of Geometry Points in a direction in which it is stretched by the transformation A Eigenspaces 83 / 127
  170. 170. Implications You can see the eigenvalues as the vector of change by the mapping T (v) = Av Therefore, for an Invertible Square Matrix A If your rank is n ⇒ if you have {v1, v2, v3, ..., vn} eigenvalues 84 / 127
  171. 171. Implications You can see the eigenvalues as the vector of change by the mapping T (v) = Av Therefore, for an Invertible Square Matrix A If your rank is n ⇒ if you have {v1, v2, v3, ..., vn} eigenvalues 84 / 127
  172. 172. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 85 / 127
  173. 173. A simple case Given a vector v ∈ V We then apply the linear transformation sequentially: v, Av, A2 v... For example A = 0.7 0.3 0.3 0.7 86 / 127
  174. 174. A simple case Given a vector v ∈ V We then apply the linear transformation sequentially: v, Av, A2 v... For example A = 0.7 0.3 0.3 0.7 86 / 127
  175. 175. We have the following sequence As you can see v = 0.5 1 , Av = 0.65 0.85 , ..., Ak v = 0.75 0.75 , ... 87 / 127
  176. 176. Geometrically We have 88 / 127
  177. 177. Notably We have that The eigenvalue λ tells whether the special vector v is stretched or shrunk or reversed or left unchanged-when it is multiplied by A. Something Notable 1 Eigenvalues can repeat!!! 2 Eigenvalues can be positive or negative 3 Eigenvalues could be 0 Properties The eigenvectors make up the nullspace of (A − λI). 89 / 127
  178. 178. Notably We have that The eigenvalue λ tells whether the special vector v is stretched or shrunk or reversed or left unchanged-when it is multiplied by A. Something Notable 1 Eigenvalues can repeat!!! 2 Eigenvalues can be positive or negative 3 Eigenvalues could be 0 Properties The eigenvectors make up the nullspace of (A − λI). 89 / 127
  179. 179. Notably We have that The eigenvalue λ tells whether the special vector v is stretched or shrunk or reversed or left unchanged-when it is multiplied by A. Something Notable 1 Eigenvalues can repeat!!! 2 Eigenvalues can be positive or negative 3 Eigenvalues could be 0 Properties The eigenvectors make up the nullspace of (A − λI). 89 / 127
  180. 180. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 90 / 127
  181. 181. An Intuition Imagine that A is a symmetric real matrix Then, we have that Av is a mapping What happens to the unitary circle? v|vT v = 1 91 / 127
  182. 182. An Intuition Imagine that A is a symmetric real matrix Then, we have that Av is a mapping What happens to the unitary circle? v|vT v = 1 91 / 127
  183. 183. We have something like A modification of the distances Unitary Circle 92 / 127
  184. 184. If we get the Q matrix We go back to the unitary circle A is a modification of distances Therefore Our best bet is to build A with specific properties at hand... 93 / 127
  185. 185. If we get the Q matrix We go back to the unitary circle A is a modification of distances Therefore Our best bet is to build A with specific properties at hand... 93 / 127
  186. 186. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 94 / 127
  187. 187. Therefore Relation with invertibility What if (A − λI) v = 0? What if v = 0? Then, columns A − λI are not linear independents. Then A − λI is not invertible... 95 / 127
  188. 188. Therefore Relation with invertibility What if (A − λI) v = 0? What if v = 0? Then, columns A − λI are not linear independents. Then A − λI is not invertible... 95 / 127
  189. 189. Therefore Relation with invertibility What if (A − λI) v = 0? What if v = 0? Then, columns A − λI are not linear independents. Then A − λI is not invertible... 95 / 127
  190. 190. Also for Determinants If A − λI is not invertible det (A − λI) = 0 ← How? Theorem A square matrix is invertible if and only if its determinant is non-zero. Proof =⇒ We know for Jordan-Gauss that an invertible matrix can be reduced to the identity by elementary matrix operations A = E1E2 · · · Ek 96 / 127
  191. 191. Also for Determinants If A − λI is not invertible det (A − λI) = 0 ← How? Theorem A square matrix is invertible if and only if its determinant is non-zero. Proof =⇒ We know for Jordan-Gauss that an invertible matrix can be reduced to the identity by elementary matrix operations A = E1E2 · · · Ek 96 / 127
  192. 192. Also for Determinants If A − λI is not invertible det (A − λI) = 0 ← How? Theorem A square matrix is invertible if and only if its determinant is non-zero. Proof =⇒ We know for Jordan-Gauss that an invertible matrix can be reduced to the identity by elementary matrix operations A = E1E2 · · · Ek 96 / 127
  193. 193. Furthermore We have then det (A) = det (E1) · · · det (Ek) An interesting thing is that, for example Let A be a K × K matrix. Let E be an elementary matrix obtained by multiplying a row of the K × K identity matrix I by a constant c = 0. Then det (E) = c. 97 / 127
  194. 194. Furthermore We have then det (A) = det (E1) · · · det (Ek) An interesting thing is that, for example Let A be a K × K matrix. Let E be an elementary matrix obtained by multiplying a row of the K × K identity matrix I by a constant c = 0. Then det (E) = c. 97 / 127
  195. 195. The same for the other elementary matrices Then, det (A) = det (E1) · · · det (Ek) = 0 Now, the return is quite simple Then, A − λI is not invertible det (A − λI) = 0 98 / 127
  196. 196. The same for the other elementary matrices Then, det (A) = det (E1) · · · det (Ek) = 0 Now, the return is quite simple Then, A − λI is not invertible det (A − λI) = 0 98 / 127
  197. 197. Now, for eigenvalues Theorem The number λ is an eigenvalue ⇐⇒ (A − λI) is not invertible i.e. singular. =⇒ The number λ is an eigenvalue ⇒ then ∃v such that (A − λI) v = 0 The columns of A − λI They are linear dependent so (A − λI) is not invertible What about ⇐=? 99 / 127
  198. 198. Now, for eigenvalues Theorem The number λ is an eigenvalue ⇐⇒ (A − λI) is not invertible i.e. singular. =⇒ The number λ is an eigenvalue ⇒ then ∃v such that (A − λI) v = 0 The columns of A − λI They are linear dependent so (A − λI) is not invertible What about ⇐=? 99 / 127
  199. 199. Now, for eigenvalues Theorem The number λ is an eigenvalue ⇐⇒ (A − λI) is not invertible i.e. singular. =⇒ The number λ is an eigenvalue ⇒ then ∃v such that (A − λI) v = 0 The columns of A − λI They are linear dependent so (A − λI) is not invertible What about ⇐=? 99 / 127
  200. 200. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 100 / 127
  201. 201. Now, How do we find eigenvalues and eigenvectors? Ok, we know that for each eigenvalue there is an eigenvector We have seen that they represent the stretching of the vectors How do we get such eigenvalues Basically, use the fact that if λ ⇒ det [A − λI] = 0 In this way We obtain a polynomial know as characteristic polynomial. 101 / 127
  202. 202. Now, How do we find eigenvalues and eigenvectors? Ok, we know that for each eigenvalue there is an eigenvector We have seen that they represent the stretching of the vectors How do we get such eigenvalues Basically, use the fact that if λ ⇒ det [A − λI] = 0 In this way We obtain a polynomial know as characteristic polynomial. 101 / 127
  203. 203. Now, How do we find eigenvalues and eigenvectors? Ok, we know that for each eigenvalue there is an eigenvector We have seen that they represent the stretching of the vectors How do we get such eigenvalues Basically, use the fact that if λ ⇒ det [A − λI] = 0 In this way We obtain a polynomial know as characteristic polynomial. 101 / 127
  204. 204. Characteristic Polynomial Then get the root of the polynomial i.e. Values of λ that make p (λ) = ao + a1λ + a2λ + · · · + anλn = 0 Then, once you have the eigenvalues For each eigenvalue λ solve (A − λI) v = 0 or Av = λv It is quite simple But a lot of theorems to get here!!! 102 / 127
  205. 205. Characteristic Polynomial Then get the root of the polynomial i.e. Values of λ that make p (λ) = ao + a1λ + a2λ + · · · + anλn = 0 Then, once you have the eigenvalues For each eigenvalue λ solve (A − λI) v = 0 or Av = λv It is quite simple But a lot of theorems to get here!!! 102 / 127
  206. 206. Characteristic Polynomial Then get the root of the polynomial i.e. Values of λ that make p (λ) = ao + a1λ + a2λ + · · · + anλn = 0 Then, once you have the eigenvalues For each eigenvalue λ solve (A − λI) v = 0 or Av = λv It is quite simple But a lot of theorems to get here!!! 102 / 127
  207. 207. Example Given A = 1 2 2 4 Find Its eigenvalues and eigenvectors. 103 / 127
  208. 208. Example Given A = 1 2 2 4 Find Its eigenvalues and eigenvectors. 103 / 127
  209. 209. Summary To solve the eigenvalue problem for an n × n matrix, follow these steps 1 Compute the determinant of A − λI. 2 Find the roots of the polynomial det (A − λI) = 0. 3 For each eigenvalue solve (A − λI) v = 0 to find the eigenvector v. 104 / 127
  210. 210. Some Remarks Something Notable If you add a row of A to another row, or exchange rows, the eigenvalues usually change. Nevertheless 1 The product of the n eigenvalues equals the determinant. 2 The sum of the n eigenvalues equals the sum of the n diagonal entries. 105 / 127
  211. 211. Some Remarks Something Notable If you add a row of A to another row, or exchange rows, the eigenvalues usually change. Nevertheless 1 The product of the n eigenvalues equals the determinant. 2 The sum of the n eigenvalues equals the sum of the n diagonal entries. 105 / 127
  212. 212. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 106 / 127
  213. 213. They impact many facets of our life!!! Example, given the composition of the linear function 107 / 127
  214. 214. Then, for recurrent systems Something like vn+1 = Avn + b Making b = 0 vn+1 = Avn The eigenvalues are telling us if the recurrent system converges or not For example if we modify the matrix A. 108 / 127
  215. 215. Then, for recurrent systems Something like vn+1 = Avn + b Making b = 0 vn+1 = Avn The eigenvalues are telling us if the recurrent system converges or not For example if we modify the matrix A. 108 / 127
  216. 216. Then, for recurrent systems Something like vn+1 = Avn + b Making b = 0 vn+1 = Avn The eigenvalues are telling us if the recurrent system converges or not For example if we modify the matrix A. 108 / 127
  217. 217. For example Here, iterations send the system to the infinity 109 / 127
  218. 218. In another Example Imagine the following example 1 F represents the number of foxes in a population 2 R represents the number of rabits in a population Then, if we have that The number of rabbits is related to the number of foxes in the following way At each time you have three times the number of rabbits minus the number of foxes 110 / 127
  219. 219. In another Example Imagine the following example 1 F represents the number of foxes in a population 2 R represents the number of rabits in a population Then, if we have that The number of rabbits is related to the number of foxes in the following way At each time you have three times the number of rabbits minus the number of foxes 110 / 127
  220. 220. Therefore We have the following relation dR dt = 3R − 1F dF dt = 1F Or as a matrix operations R F = 3 −1 0 1 R F 111 / 127
  221. 221. Therefore We have the following relation dR dt = 3R − 1F dF dt = 1F Or as a matrix operations R F = 3 −1 0 1 R F 111 / 127
  222. 222. Geometrically As you can see through the eigenvalues we have a stable population Rabbits Foxs 112 / 127
  223. 223. Therefore We can try to cast our problems as system of equations Solve by methods found in linear algebra Then, using properties of the eigenvectors We can look at sought properties that we would like to have 113 / 127
  224. 224. Therefore We can try to cast our problems as system of equations Solve by methods found in linear algebra Then, using properties of the eigenvectors We can look at sought properties that we would like to have 113 / 127
  225. 225. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 114 / 127
  226. 226. Assume a matrix A Definition An n × n matrix A is diagonalizable is called diagonalizable if there exists an invertible matrix P such that P−1AP is a diagonal matrix. Some remarks Is every diagonalizable matrix invertible? 115 / 127
  227. 227. Assume a matrix A Definition An n × n matrix A is diagonalizable is called diagonalizable if there exists an invertible matrix P such that P−1AP is a diagonal matrix. Some remarks Is every diagonalizable matrix invertible? 115 / 127
  228. 228. Nope Given the structure P−1 AP =       λ1 0 · · · 0 0 λ2 · · · 0 ... ... ... ... 0 0 · · · λn       Then using the determinant det P−1 AP = det [P]−1 det [A] det [P] = det [A] = n i= λi if one of the eigenvalues of A is zero The determinant of A is zero, and hence A is not invertible. 116 / 127
  229. 229. Nope Given the structure P−1 AP =       λ1 0 · · · 0 0 λ2 · · · 0 ... ... ... ... 0 0 · · · λn       Then using the determinant det P−1 AP = det [P]−1 det [A] det [P] = det [A] = n i= λi if one of the eigenvalues of A is zero The determinant of A is zero, and hence A is not invertible. 116 / 127
  230. 230. Nope Given the structure P−1 AP =       λ1 0 · · · 0 0 λ2 · · · 0 ... ... ... ... 0 0 · · · λn       Then using the determinant det P−1 AP = det [P]−1 det [A] det [P] = det [A] = n i= λi if one of the eigenvalues of A is zero The determinant of A is zero, and hence A is not invertible. 116 / 127
  231. 231. Actually Theorem A diagonal matrix is invertible if and only if its eigenvalues are nonzero. Is Every Invertible Matrix Diagonalizable? Consider the matrix: A = 1 1 0 1 The determinant of A is 1, hence A is invertible (Characteristic Polynomial) p (λ) = det [A − λI] = (1 − t)2 117 / 127
  232. 232. Actually Theorem A diagonal matrix is invertible if and only if its eigenvalues are nonzero. Is Every Invertible Matrix Diagonalizable? Consider the matrix: A = 1 1 0 1 The determinant of A is 1, hence A is invertible (Characteristic Polynomial) p (λ) = det [A − λI] = (1 − t)2 117 / 127
  233. 233. Actually Theorem A diagonal matrix is invertible if and only if its eigenvalues are nonzero. Is Every Invertible Matrix Diagonalizable? Consider the matrix: A = 1 1 0 1 The determinant of A is 1, hence A is invertible (Characteristic Polynomial) p (λ) = det [A − λI] = (1 − t)2 117 / 127
  234. 234. Therefore, you have a repetition in the eigenvalue Thus, the geometric multiplicity of the eigenvalue 1 is 1, 1 0 T Since the geometric multiplicity is strictly less than the algebraic multiplicity, the matrix A is defective and not diagonalizable. Why? Let us to look at the eigenvectors for this answer 118 / 127
  235. 235. Therefore, you have a repetition in the eigenvalue Thus, the geometric multiplicity of the eigenvalue 1 is 1, 1 0 T Since the geometric multiplicity is strictly less than the algebraic multiplicity, the matrix A is defective and not diagonalizable. Why? Let us to look at the eigenvectors for this answer 118 / 127
  236. 236. Relation with Eigenvectors Suppose that the n × n matrix A has n linearly independent eigenvectors v1, v2, ..., vn Put them into an eigenvector matrix P P = v1 v2 ... vn 119 / 127
  237. 237. Relation with Eigenvectors Suppose that the n × n matrix A has n linearly independent eigenvectors v1, v2, ..., vn Put them into an eigenvector matrix P P = v1 v2 ... vn 119 / 127
  238. 238. We have What if we apply it to the canonical basis elements? P (ei) = vi Then apply this to the matrix A AP (ei) = λivi Finally P−1 AP (ei) = λiei 120 / 127
  239. 239. We have What if we apply it to the canonical basis elements? P (ei) = vi Then apply this to the matrix A AP (ei) = λivi Finally P−1 AP (ei) = λiei 120 / 127
  240. 240. We have What if we apply it to the canonical basis elements? P (ei) = vi Then apply this to the matrix A AP (ei) = λivi Finally P−1 AP (ei) = λiei 120 / 127
  241. 241. Therefore ei is the set of eigenvectors of P−1 AP I = e1 e2 · · · en Then P−1 AP = P−1 API = λ1e1 λ2e2 · · · λnen 121 / 127
  242. 242. Therefore ei is the set of eigenvectors of P−1 AP I = e1 e2 · · · en Then P−1 AP = P−1 API = λ1e1 λ2e2 · · · λnen 121 / 127
  243. 243. Therefore We have that P−1 AP =        λ1 0 · · · 0 0 λ2 0 ... ... 0 ... 0 0 · · · 0 λn        = D 122 / 127
  244. 244. Therefore We can see the diagonalization as a decomposition A P P−1 AP = IDP In a similar way A = PDP−1 Therefore Only if we have n linearly independent eigenvectors (Different Eigenvalues), we can diagonalize it. 123 / 127
  245. 245. Therefore We can see the diagonalization as a decomposition A P P−1 AP = IDP In a similar way A = PDP−1 Therefore Only if we have n linearly independent eigenvectors (Different Eigenvalues), we can diagonalize it. 123 / 127
  246. 246. Therefore We can see the diagonalization as a decomposition A P P−1 AP = IDP In a similar way A = PDP−1 Therefore Only if we have n linearly independent eigenvectors (Different Eigenvalues), we can diagonalize it. 123 / 127
  247. 247. Outline 1 Orthonormal Basis Introduction The Norm The Row Space and Nullspace are Orthogonal sub-spaces inside Rn Orthogonal Complements Fundamental Theorems of Linear Algebra Projections Projection Onto a Subspace Orthogonal Bases and Gram-Schmidt Solving a Least Squared Error The Gram Schmidt Process The Gram Schmidt Algorithm and the QR Factorization 2 Eigenvectors Introduction What are eigenvector good for? Modification on Distances Relation with Invertibility Finding Eigenvalues and Eigenvectors Implications of Existence of Eigenvalues Diagonalization of Matrices Interesting Derivations 124 / 127
  248. 248. Some Interesting Properties What is A2 Assuming n × n matrix that can be diagonlized. Quite simple Ak = SΛK S−1 What happens if for all |λi| < 1 Ak → 0 when k −→ ∞ 125 / 127
  249. 249. Some Interesting Properties What is A2 Assuming n × n matrix that can be diagonlized. Quite simple Ak = SΛK S−1 What happens if for all |λi| < 1 Ak → 0 when k −→ ∞ 125 / 127
  250. 250. Some Interesting Properties What is A2 Assuming n × n matrix that can be diagonlized. Quite simple Ak = SΛK S−1 What happens if for all |λi| < 1 Ak → 0 when k −→ ∞ 125 / 127
  251. 251. Some Basic Properties of the Symmetric Matrices Symmetric Matrix 1 A symmetric matrix has only real eigenvalues. 2 The eigenvectors can be chosen orthonormal. 126 / 127
  252. 252. Spectral Theorem Theorem Every symmetric matrix has the factorization A = QΛQT with the real eigenvalues in Λ and orthonormal eigenvectors P = Q. Proof A direct proof from the previous ideas. 127 / 127
  253. 253. Spectral Theorem Theorem Every symmetric matrix has the factorization A = QΛQT with the real eigenvalues in Λ and orthonormal eigenvectors P = Q. Proof A direct proof from the previous ideas. 127 / 127

×