Appendix A
Matrix Algebra
A.l_ Definitions
Ann x m matrix is a rectangular array of numbers with n rows and m columns. For
example,
5 -4 0
-1 21
isa 2x3 matrix. The numbers n and m are the dimensions of the matrix. A matrix
|nsch that the dimensions n and m are equal is called a square matrix. A column
vector is an n x 1 matrix and a row vector is a 1 x m matrix.
| ‘Two matrices of the same size can be added by the addition of their corresponding
|dements. :
‘
Example A.1 The sum of two matrices.
The sum of the 2 x 3 matrices
s 5-4 0 a (24 8
-1 21) ™ 3 0.-5
419420 APPENDIX A. MATRIX ALGEBRA
Only matrices that have exactly the same dimensions can be added.
The scalar product of a constant with a matrix is formed by multiplying every
entry in the matrix by the constant.
Example A.2 The scalar product of a constant and a matriz.
The scalar product of the constant 3 with the matrix
30 3
22-4
3(3.9 3\_(9 0 9
22 -4)°\6 6 -12)°
The dot product of the row vector
is
(1 «+. tn)
with the column vector
wu
Un
is the number zy +... + 2nYn-
‘The product of an n x m matrix with an m x r matrix is an n x r matrix, where
the entry in row i and column j is defined to be the dot product of row i in the first
matrix with column j in the second matrix.
Example A.3 The product of two matrices.
The product of the 2 x 3 matrix
) 30° 3
22 -4
ist
for
but}A.1. DEFINITIONS
with the 3 x 2 matrix
is the 2 x 2 matrix
o
G2 i) soe (3%)
22-4)(5 0.0
The entries are obtained as follows:
Row | Column Entry
dyad (3)(1) + (0)(8) + (8)(2)
1 2 (3)(~2) + (0)(0) + (3)(-1)
2 1 (2)(1) + (2)(8) +-(~4)(2
2 2 (2)(=2) + (2)(0)(—4)(—1
Unlike ordinary multiplication, matrix multiplication is usually not commutative;
for example,
(22 a) 3 -(3 7%)
22 -4)\> 9 0 0
1 -2\,, 1-4
rs (32 3)-[ 2 0 9].
2-1 4-2 10
Matrices must have compatible sizes to be multiplied; the number of columns in
the first matrix must be the same as the number of rows in the second matrix. The
product has the number of rows of the first matrix and the number of columns of the
second matrix.
The transpose of a matrix is obtained by interchanging rows and columns.
Example A.4 The transpose of a matric.
The transpose of the matrix,422 APPENDIX A. MATRIX ALGEBRA
)
The determinant of an n x n matrix is a number that can be computed from the
matrix. The determinant of a 2 x 2 matrix is computed by the formula
aw(¢ b) soda
The determinant of a 3 x 3 matrix is given by
is the matrix
wenn
= aei + bfg + cdh — afh — bdi —ceg.
ao
ab
det} d e
gh
Example A.5 The determinant of a2 x 2 matriz.
au ( cn ) = (2)(-1) - (1)(3) =-2-3 = -5.
Example A.6 The determinant of a3 x 3 matriz.
2-4-24 (1)(-1)(8) +0- 2(-1) — (2)(-1)(-1) -1-2-2-0-4-3
= 16-34+0-2-4+0=7.
Although similar formulas exist for computing the determinants for matrices which
are larger than 2 x 2 or 3 x 3, they are quite complicated and we will not discuss them
here. (The formulas for determinants of 4 x 4 and 5 x 5 matrices have 24 and 120A.1. DEFINITIONS 423,
terms, respectively.) It is usually most convenient to use computer software to find the
determinant of a large matrix.
‘A special type of matrix is the identity matrix. An identity matrix is a square
matrix that contains 1s on its diagonal and 0 everywhere else. If I is an identity matrix
and A is an arbitrary matrix, then
as long as the products AI and IA are defined.
Example A.7 Identity matrices.
The matrices
orcs
Hooo
on
no
SS
—
cou
ono
noo
_S
a
coon
ooHro
are identity matrices.
‘A matrix A~? is defined to be the inverse of the n x n matrix A if
AA = AtAST
where J is the n x n identity matrix. For computing the inverse of a 2 x 2 matrix, the
formula 4
@ b\7_ 1 a -b
cd} “ad-b\-c a
is used. The formula : .
ab c\" ,(ei-fh ch-bi bf-ce
die f| = Z| fo-di.ai-eg cd—af
gh dh—eg bg—ah ae—bd
where ;
D.=aei + bfg + cdh — afh —bdi —ceg
is used for computing the inverse of a3 x 3 matrix,424 APPENDIX A. MATRIX ALGEBRA
Example A.8 The inverse of a2 x 2 matric.
Example A.9 The inverse of a3 x 3 matrix:
300 1-1-2-0. 0-0-0-1 6-2-0-1
iio %=-2)-1-1 3+1-0(-2) 0-1-3.2
201 1-0-1(-2) 0(-2)~3-0 3-1-0-1
where
D=3-1-140-2-(-2)+0:1-0-3-2:040-1-140-1(-2)
so
3.0 0\7 1f-10 0 1/3 0 0
e212] =3/ -5 3 -6 ]=| -5/3 1 -2
-2-0 1 20 3 23-0 1
As in the case of determinants, we will not compute the inverses of matrices that
are larget than 3 x 3. Computer software such as SAS (see Appendix B) can be used
to compute the inverses of larger matrices. However, there is a special case where it is
easy to compute the inverses of matrices of larger dimension. A matrix such that all
entries off the main diagonal are zero is called a diagonal matrix. The inverse of a
diagonal matrix is given by the following formula:
a4 0 ...°0\7 Ya 0 .. 0
0 a... 0 0 IfaA.2,. EIGENVALUES AND EIGENVECTORS
_ A.2- Eigenvalues and Eigenvectors
In many areas of mathematics and science, one wishes to find all the numbers 4 and
all vectors v that satisfy the equation
Av=dv (Aa)
where A is a square matrix. Any number Satisfying (A.1) is called an eigenvalue of
A. A vector v satisfying (A.1) is called an eigenvector of A. The eigenvalues 4 are
found by solving the equation’
det(A — AI) = 0. (A2)
‘Then for every ) that is found as a solution of (A.2), the corresponding set of eigen-
vectors v is found from (A.1).
; Example A.10 Finding eigenvalues and eigenvectors.
| Find the eigenvalues and eigenvectors of the matrix
52
22;°
Solving (A.2) gives us :
ael(2 2)oC0 iy) = #C2? 223)
= (5-A)(2—2) — (2)(2)
= Y-T4+6=0.
Solving this equation for \ gives 4 = 6 and 2 = 1 for eigenvalues:
Substituting A; = 6 into (A.1) gives
= GGG)
This leads to the system|of equations
4
ba +2y
Qn + Qy426 APPENDIX A. MATRIX ALGEBRA
or
-r+24y = 0
2n—4y = 0,
which implies that z =.2y. Thus the eigenvectors corresponding to \ = 6 are
2y
n= :
(3)
for arbitrary y, which lie on a line through the origin of slope 1/2.
Substituting \2 = 1 into (A.1) gives .
where
This leads to the system of equations
Se+Qy "= x
Qet+2y = y
or
4e+2y = 0
Wty = 0,
which implies that y= —22. Therefore the eigenvectors corresponding to Ay = 1 are
a=(.2):
which lie on a line through the origin of slope —2.
In Example A.10, the lines with slopes 1/2 and —2 are perpendicular to each other.
It can be shown in genetal that any two eigenvectors that correspond to unique eigen.
values are perpendicular. Computer software that finds eigenvalues and eigenvectors
usually gives a set of mutually perpendicular eigenvectors—one corresponding to eachft
4.2. EIGENVALUES AND EIGENVECTORS 427
igenvaluié—such that o?v = 1 for each eigenvector v. Corresponding to the eigenvalue
6, the SAS (Statistical Software System) software package gives the eigenvector
0.894
0.447 )*
0.447
A. 0.894
as the eigenvector corresponding to the eigenvalue 1. See Section B.5 for more details.
| It gives
Example A.11 Repeated eigenvalues.
Find the eigenvalues and eigenvectors of the matrix
| oop (HE)
Equation (A.2) becomes
9 6 3 100 9-r 6
det|{ 6 4 2 ]-A] 0 1 0 |}=det| 6 4-A
3.21 001 3 2
which is
(9—d)(4—d)(1- A) +6-2-3-4+3-6-2—4(9— d) —36(1- A) -9(4—A) = (14-A)2? = 0.
wae
VRo
Hw
‘The solutions to this equation are 14, 0, and 0, which are the eigenvalues.
Substituting 1, = 14 into (A.1) gives the system of equations
Or+6y+3z = lz
6c+4y+2z = ldy
Bc+2Qytiz = Ide
or
\ -be+6yt+az = 0 |
} 6z-1y+2z = 0
3a+2y-13z2 = 0bea!
‘
428 APPENDIX A. MATRIX ALGEBRA
which has as its solution x = 3z and y = 2z where z is arbitrary. Thus the eigenvectors
corresponding to A; = 14 are of the form
3z
uy=| 2z,
z
When d= 0, (A.1) becomes
90 + 6y +32
6x + dy + 22
3x4 2y +z
(2/3)y — (1/3)z where y and z are arbitrary, or
—(2/3)y - (1/3)2
y
‘Whereas the set of eigenvectors corresponding to \; = 14 is a line through’ the
origin, the set of eigenvectors corresponding to Az = A3 = 0 is a plane through the
origin. This is because 0 is a solution of multiplicity 2 of (A.2). In general, a solution
of multiplicity k of (A.2) produces a k-dimensional hyperplane of eigenvectors through
the origin.
SAS gives the eigenvector
0.801
. wy =| 0.535
0.267
corresponding to.\ = 14 and the eigenvectors
-0.365 -0.473 5
m=| 0.084 | and v=] 0841
0.927 0.262
corresponding to 2 = Ag = 0. The perpendicular eigenvectors vp and vy determine a
plaite. See Appendix B for details of using SAS to compute eigenvalues.
An important application of eigenvectors in pattern recognition is to find the direc-
tions of greatest variance of a covariance matrix. The eigenvector corresponding to the
largest eigenvalue bf the covariance matrix 3 is the direction of greatest variance. The
largest eigenvalue is the variance in that direction. The eigenvector corresponding to
the second largest eigenvector is the direction of greatest variation perpendicular to theA.2, EIGENVALUES AND EIGENVECTORS 429
first eigenvector. In general, the kth eigenvector is the direction of greatest variation
perpendicular to the first through (k —1)st eigenvectors. The eigenvalues of 5, also
called principal components, allow us to identify the directions of greatest variation
in a class. They can help identify new features that are useful for classification.