Linear Algebra Demystified

PREFACE

This book is for people who want to get a head start and learn the basic concepts of linear algebra. Suitable for self-study or as a reference that puts solving problems within easy reach, this book can be used by students or professionals looking for a quick refresher. If you’re looking for a simplified presentation with explicitly solved problems for self-study, this book will help you. If you’re a student taking linear algebra and need an informative aid to keep you ahead of the game, this book is the perfect supplement to the classroom.

The topics covered fit those usually taught in a one-semester undergraduate course, but the book is also useful to graduate students as a quick refresher. The book can serve as a good jumping off point for students to read before taking a course. The presentation is informal and the emphasis is on showing students how to solve problems that are similar to those they are likely to encounter in homework and examinations. Enhanced detail is used to uncover techniques used to solve problems rather than leaving the how and why of homework solutions a secret.

While linear algebra begins with the solution of systems of linear equations, it quickly jumps off into abstract topics like vector spaces, linear transformations, determinants, and solving eigenvector problems. Many students have a hard time struggling through these topics. If you are having a hard time getting through your courses because you don’t know how to solve problems, this book should help you make progress.

As part of a self-study course, this book is a good place to get a first exposure to the subject or it is a good refresher if you’ve been out of school for a long time. After reading and doing the exercises in this book it will be much easier for you to tackle standard linear algebra textbooks or to move on to a more advanced treatment.

The organization of the book is as follows. We begin with a discussion of solution techniques for solving linear systems of equations. After introducing the notion of matrices, we illustrate basic matrix algebra operations and techniques such as finding the transpose of a matrix or computing the trace. Next we study determinants, vectors, and vector spaces. This is followed by the study of linear transformations. We then devote some time showing how to find the eigenvalues and eigenvectors of a matrix. This is followed by a chapter that discusses several special types of matrices that are important. This includes symmetric, Hermitian, orthogonal, and unitary matrices. We finish the book with a review of matrix decompositions, specifically LU, SVD, and QR decompositions.

Each chapter has several examples that are solved in detail. The idea is to remove the mystery and show the student how to solve problems. Exercises at the end of each chapter have been designed to correspond to the solved problems in the text so that the student can reinforce ideas learned while reading the chapter. A final exam, with similar questions, at the end of the book gives the student a chance to reinforce these notions after completing the text.

David McMahon

CHAPTER

Systems of Linear Equations

A linear equation with n unknowns is an equation of the type

a1x1 + a2x2 + … + anxn = b

In many situations, we are presented with m linear equations in n unknowns. Such a set is known as a system of linear equations and takes the form

The terms x1, x2, … , xn are the unknowns or variables of the system, while the aij are called coefficients. The bi on the right-hand side are fixed numbers or scalars. The goal is to find the values of the x1, x2, … , xn such that the equations are satisfied.

EXAMPLE 1-1

Consider the system

3x + 2y − z = 7

       4x + 9y = 2

x + 5y − 3z = 0

Does (x, y, z) = (2, 1, 1) solve the system? What about ?

SOLUTION 1-1

We substitute the values of (x, y, z) into each equation. Trying (x, y, z) = (2, 1, 1) in the first equation, we obtain

3 (2) + 2 (1) − 1 = 6 + 2 − 1 = 7

and so the first equation is satisfied. Using the substitution in the second equation, we find

4 (2) + 9 (1) = 8 + 9 = 17 ≠ 2

The second equation is not satisfied; therefore, (x, y, z) = (2, 1, 1) cannot be a solution to this system of equations.

Now we try the second set of numbers . Substitution in the first equation gives

Again, the first equation is satisfied. Trying the second equation gives

This time the second equation is also satisfied. Finally, the third equation works out to be

This shows that the third equation is satisfied as well. Therefore we conclude that

is a solution to the system.

Consistent and Inconsistent Systems

When at least one solution exists for a given system of linear equations, we call that system consistent. If no solution exists, the system is called inconsistent. The solution to a system is not necessarily unique. A consistent system either has a unique solution or it can have an infinite number of solutions. We summarize these ideas in Fig. 1-1.

Fig. 1-1. Description of solution possibilities.

If a consistent system has an infinite number of solutions, if we can define a solution in terms of some extra parameter t, we call this a parametric solution.

Matrix Representation of a System of Equations

It is convenient to write down the coefficients and scalars in a linear system of equations as a rectangular array of numbers called a matrix. Each row in the array corresponds to one equation. For a system with m equations in n unknowns, there will be m rows in the matrix.

The array will have n + 1 columns. Each of the first n columns is used to write the coefficients that multiply each of the unknown variables. The last column is used to write the numbers found on the right-hand side of the equations. Consider the set of equations used in the last example:

3x + 2y − z = 7

     4x + 9y = 2

x + 5y − 3z = 0

The matrix used to represent this system is

We represent this set of equations

2x + y = −7

x − 5y = 12

by the matrix

One way we can characterize a matrix is by the number of rows and columns it has. A matrix with m rows and n columns is referred to as an m × n matrix. Sometimes matrices are square, meaning that the number of rows equals the number of columns.

We refer to a given element found in a matrix by identifying its row and column position. This can be done using the notation (i, j) to refer to the element located at row i and column j. Rows are numbered starting with 1 at the top of the matrix, increasing as we move down the matrix. Columns are numbered starting with 1 on the left-hand side.

An alternative method of identifying elements in a matrix is to use a subscript notation. Matrices are often identified with italicized or bold capital letters. So A, B, C or A, B, C can be used as labels to identify matrices. The corresponding small letter is then used to identify individual elements of the matrix, with subscripts indicating the row and column where the term is located. For a matrix A, we can use aij to identify the element located at the row and column position (i, j).

As an example, consider the 3 × 4 matrix

The element located at row 2 and column 3 of this matrix can be indicated by writing (2, 3) or b23. This number is

b23 = −1

The element located at row 3 and column 2 is

b32 = 17

The subscript notation is shown in Fig. 1-2.

Fig. 1-2. The indexing of an element found at row i and column j of a matrix.

A matrix that includes the entire linear system is called an augmented matrix. We can also make a matrix that is made up only of the coefficients that multiply the unknown variables. This is known as the coefficient matrix. For the system

5x − y + 9z = 2

4x + 2y − z = 18

x + y + 3z = 6

the coefficient matrix is

We can find a solution to a linear system of equations by applying a set of elementary operations to the augmented matrix.

Solving a System Using Elementary Operations

There exist three elementary operations that can be applied to a system of linear equations without fundamentally changing that system. These are

•   Exchange two rows of the matrix.

•   Replace a row by a scalar multiple of itself, as long as the scalar is nonzero.

•   Replace one row by adding the scalar multiple of another row.

Let’s introduce some shorthand notation to describe these operations and demonstrate using the matrix

To indicate the exchange of rows 2 and 3, we write

R2 R3

This transforms the matrix as follows:

Now let’s consider the operation where we replace a row by a scalar multiple of itself. Let’s say we wanted to replace the first row in the following way:

2R1 → R1

The matrix would be transformed as

In the third type of operation, we replace a selected row by adding a scalar multiple of a different row. Consider

−2R2 + R1 → R1

The matrix becomes

The solution to the system is obtained when this set of operations brings the matrix into triangular form. This type of elimination is sometimes known as Gaussian elimination.

Triangular Matrices

Generally, the goal of performing the elementary operations on a system is to get it in a triangular form. A system that is in an upper triangular form is

This augmented matrix represents the equations

5x − y + z = 11

    2y − z = 2

            3z