Appendix A Matrix Algebra A.l_ Definitions Ann x m matrix is a rectangular array of numbers with n rows and m columns. For example, 5 -4 0 -1 21 isa 2x3 matrix. The numbers n and m are the dimensions of the matrix. A matrix |nsch that the dimensions n and m are equal is called a square matrix. A column vector is an n x 1 matrix and a row vector is a 1 x m matrix. | ‘Two matrices of the same size can be added by the addition of their corresponding |dements. : ‘ Example A.1 The sum of two matrices. The sum of the 2 x 3 matrices s 5-4 0 a (24 8 -1 21) ™ 3 0.-5 419420 APPENDIX A. MATRIX ALGEBRA Only matrices that have exactly the same dimensions can be added. The scalar product of a constant with a matrix is formed by multiplying every entry in the matrix by the constant. Example A.2 The scalar product of a constant and a matriz. The scalar product of the constant 3 with the matrix 30 3 22-4 3(3.9 3\_(9 0 9 22 -4)°\6 6 -12)° The dot product of the row vector is (1 «+. tn) with the column vector wu Un is the number zy +... + 2nYn- ‘The product of an n x m matrix with an m x r matrix is an n x r matrix, where the entry in row i and column j is defined to be the dot product of row i in the first matrix with column j in the second matrix. Example A.3 The product of two matrices. The product of the 2 x 3 matrix ) 30° 3 22 -4 ist for but}A.1. DEFINITIONS with the 3 x 2 matrix is the 2 x 2 matrix o G2 i) soe (3%) 22-4)(5 0.0 The entries are obtained as follows: Row | Column Entry dyad (3)(1) + (0)(8) + (8)(2) 1 2 (3)(~2) + (0)(0) + (3)(-1) 2 1 (2)(1) + (2)(8) +-(~4)(2 2 2 (2)(=2) + (2)(0)(—4)(—1 Unlike ordinary multiplication, matrix multiplication is usually not commutative; for example, (22 a) 3 -(3 7%) 22 -4)\> 9 0 0 1 -2\,, 1-4 rs (32 3)-[ 2 0 9]. 2-1 4-2 10 Matrices must have compatible sizes to be multiplied; the number of columns in the first matrix must be the same as the number of rows in the second matrix. The product has the number of rows of the first matrix and the number of columns of the second matrix. The transpose of a matrix is obtained by interchanging rows and columns. Example A.4 The transpose of a matric. The transpose of the matrix,422 APPENDIX A. MATRIX ALGEBRA ) The determinant of an n x n matrix is a number that can be computed from the matrix. The determinant of a 2 x 2 matrix is computed by the formula aw(¢ b) soda The determinant of a 3 x 3 matrix is given by is the matrix wenn = aei + bfg + cdh — afh — bdi —ceg. ao ab det} d e gh Example A.5 The determinant of a2 x 2 matriz. au ( cn ) = (2)(-1) - (1)(3) =-2-3 = -5. Example A.6 The determinant of a3 x 3 matriz. 2-4-24 (1)(-1)(8) +0- 2(-1) — (2)(-1)(-1) -1-2-2-0-4-3 = 16-34+0-2-4+0=7. Although similar formulas exist for computing the determinants for matrices which are larger than 2 x 2 or 3 x 3, they are quite complicated and we will not discuss them here. (The formulas for determinants of 4 x 4 and 5 x 5 matrices have 24 and 120A.1. DEFINITIONS 423, terms, respectively.) It is usually most convenient to use computer software to find the determinant of a large matrix. ‘A special type of matrix is the identity matrix. An identity matrix is a square matrix that contains 1s on its diagonal and 0 everywhere else. If I is an identity matrix and A is an arbitrary matrix, then as long as the products AI and IA are defined. Example A.7 Identity matrices. The matrices orcs Hooo on no SS — cou ono noo _S a coon ooHro are identity matrices. ‘A matrix A~? is defined to be the inverse of the n x n matrix A if AA = AtAST where J is the n x n identity matrix. For computing the inverse of a 2 x 2 matrix, the formula 4 @ b\7_ 1 a -b cd} “ad-b\-c a is used. The formula : . ab c\" ,(ei-fh ch-bi bf-ce die f| = Z| fo-di.ai-eg cd—af gh dh—eg bg—ah ae—bd where ; D.=aei + bfg + cdh — afh —bdi —ceg is used for computing the inverse of a3 x 3 matrix,424 APPENDIX A. MATRIX ALGEBRA Example A.8 The inverse of a2 x 2 matric. Example A.9 The inverse of a3 x 3 matrix: 300 1-1-2-0. 0-0-0-1 6-2-0-1 iio %=-2)-1-1 3+1-0(-2) 0-1-3.2 201 1-0-1(-2) 0(-2)~3-0 3-1-0-1 where D=3-1-140-2-(-2)+0:1-0-3-2:040-1-140-1(-2) so 3.0 0\7 1f-10 0 1/3 0 0 e212] =3/ -5 3 -6 ]=| -5/3 1 -2 -2-0 1 20 3 23-0 1 As in the case of determinants, we will not compute the inverses of matrices that are larget than 3 x 3. Computer software such as SAS (see Appendix B) can be used to compute the inverses of larger matrices. However, there is a special case where it is easy to compute the inverses of matrices of larger dimension. A matrix such that all entries off the main diagonal are zero is called a diagonal matrix. The inverse of a diagonal matrix is given by the following formula: a4 0 ...°0\7 Ya 0 .. 0 0 a... 0 0 IfaA.2,. EIGENVALUES AND EIGENVECTORS _ A.2- Eigenvalues and Eigenvectors In many areas of mathematics and science, one wishes to find all the numbers 4 and all vectors v that satisfy the equation Av=dv (Aa) where A is a square matrix. Any number Satisfying (A.1) is called an eigenvalue of A. A vector v satisfying (A.1) is called an eigenvector of A. The eigenvalues 4 are found by solving the equation’ det(A — AI) = 0. (A2) ‘Then for every ) that is found as a solution of (A.2), the corresponding set of eigen- vectors v is found from (A.1). ; Example A.10 Finding eigenvalues and eigenvectors. | Find the eigenvalues and eigenvectors of the matrix 52 22;° Solving (A.2) gives us : ael(2 2)oC0 iy) = #C2? 223) = (5-A)(2—2) — (2)(2) = Y-T4+6=0. Solving this equation for \ gives 4 = 6 and 2 = 1 for eigenvalues: Substituting A; = 6 into (A.1) gives = GGG) This leads to the system|of equations 4 ba +2y Qn + Qy426 APPENDIX A. MATRIX ALGEBRA or -r+24y = 0 2n—4y = 0, which implies that z =.2y. Thus the eigenvectors corresponding to \ = 6 are 2y n= : (3) for arbitrary y, which lie on a line through the origin of slope 1/2. Substituting \2 = 1 into (A.1) gives . where This leads to the system of equations Se+Qy "= x Qet+2y = y or 4e+2y = 0 Wty = 0, which implies that y= —22. Therefore the eigenvectors corresponding to Ay = 1 are a=(.2): which lie on a line through the origin of slope —2. In Example A.10, the lines with slopes 1/2 and —2 are perpendicular to each other. It can be shown in genetal that any two eigenvectors that correspond to unique eigen. values are perpendicular. Computer software that finds eigenvalues and eigenvectors usually gives a set of mutually perpendicular eigenvectors—one corresponding to eachft 4.2. EIGENVALUES AND EIGENVECTORS 427 igenvaluié—such that o?v = 1 for each eigenvector v. Corresponding to the eigenvalue 6, the SAS (Statistical Software System) software package gives the eigenvector 0.894 0.447 )* 0.447 A. 0.894 as the eigenvector corresponding to the eigenvalue 1. See Section B.5 for more details. | It gives Example A.11 Repeated eigenvalues. Find the eigenvalues and eigenvectors of the matrix | oop (HE) Equation (A.2) becomes 9 6 3 100 9-r 6 det|{ 6 4 2 ]-A] 0 1 0 |}=det| 6 4-A 3.21 001 3 2 which is (9—d)(4—d)(1- A) +6-2-3-4+3-6-2—4(9— d) —36(1- A) -9(4—A) = (14-A)2? = 0. wae VRo Hw ‘The solutions to this equation are 14, 0, and 0, which are the eigenvalues. Substituting 1, = 14 into (A.1) gives the system of equations Or+6y+3z = lz 6c+4y+2z = ldy Bc+2Qytiz = Ide or \ -be+6yt+az = 0 | } 6z-1y+2z = 0 3a+2y-13z2 = 0bea! ‘ 428 APPENDIX A. MATRIX ALGEBRA which has as its solution x = 3z and y = 2z where z is arbitrary. Thus the eigenvectors corresponding to A; = 14 are of the form 3z uy=| 2z, z When d= 0, (A.1) becomes 90 + 6y +32 6x + dy + 22 3x4 2y +z (2/3)y — (1/3)z where y and z are arbitrary, or —(2/3)y - (1/3)2 y ‘Whereas the set of eigenvectors corresponding to \; = 14 is a line through’ the origin, the set of eigenvectors corresponding to Az = A3 = 0 is a plane through the origin. This is because 0 is a solution of multiplicity 2 of (A.2). In general, a solution of multiplicity k of (A.2) produces a k-dimensional hyperplane of eigenvectors through the origin. SAS gives the eigenvector 0.801 . wy =| 0.535 0.267 corresponding to.\ = 14 and the eigenvectors -0.365 -0.473 5 m=| 0.084 | and v=] 0841 0.927 0.262 corresponding to 2 = Ag = 0. The perpendicular eigenvectors vp and vy determine a plaite. See Appendix B for details of using SAS to compute eigenvalues. An important application of eigenvectors in pattern recognition is to find the direc- tions of greatest variance of a covariance matrix. The eigenvector corresponding to the largest eigenvalue bf the covariance matrix 3 is the direction of greatest variance. The largest eigenvalue is the variance in that direction. The eigenvector corresponding to the second largest eigenvector is the direction of greatest variation perpendicular to theA.2, EIGENVALUES AND EIGENVECTORS 429 first eigenvector. In general, the kth eigenvector is the direction of greatest variation perpendicular to the first through (k —1)st eigenvectors. The eigenvalues of 5, also called principal components, allow us to identify the directions of greatest variation in a class. They can help identify new features that are useful for classification.