David Pool

Linear Algebra
LINEAR ALGEBRA
Math 211
Course Outline
Jim Hartman
Taylor 305
Ext. 2239
Semester I, 2004
The College of Wooster
Wooster, OH 44691
Linear algebra is one of the most useful courses a student of science or

mathematics will ever take. It is the first course where concepts are at least
as important as calculations, and applications are motivating and mind
opening.
Applications of linear algebra to science and real life are numerous. The
solutions to many problems in physics, engineering, biology, chemistry,
medicine, computer graphics, image processing, economics, and sociology
require tools from linear algebra. So do all main branches of modern
mathematics.
Resources
Policies
Grades
Goals
Academic Integrity
DUE DATES
David Poole, LINEAR ALGEBRA: A Modern Introduction

Maple 9.5 will be used extensively to perform routine computations. Some homework
and exam questions will be best completed using Maple.
November 4: Draft 1 of Proof 3

November 10: Draft 1 of Research Paper
November 12: Exam 2
November 17: Draft 2 of Proof 3
November 22: Draft 2 of Research Paper
December 3: Final Draft Proofs 4 & 5
December 10: Final Draft Matrix Project
Office Hours
http://www.wooster.edu/math/linalg/ (1 of 3)2005/03/08 03:48:53 .
Linear Algebra
Monday
Tuesday
Wednesday
Thursday
Friday
2-4pm
10-11am, 2-5pm
2-4pm
By Appointment Only
9-10am, 2-3pm
Tentative Lecture Schedule

Course Notes and Materials
Short History of Linear Algebra

Writing Assignments
"Windows" program for matrix arithmetic
Sample Exams
Exam 1 Fall 2004 Solutions(pdf)
Review for Exam 1
Exam 2 Fall 2004 Solutions(pdf)
Review for Exam 2
Lecture Notes
Maple Files
Iterative Methods for Solving Systems of Equations
Maple Lab#1 (PDF Version)
Maple Lab#2 (PDF Version)
Matrix Inverse Algorithm
Diagonalizing Matrices
Maple Command Sheet
Matrix Project Example
Linear Transformation Movie
Change of Basis
Definitions
Facts
Iterative Methods for Finding Eigenvalues (PDF version)
Fundamental Theorem of Invertible Matrices
Diagonalizability of Matrices
A Diagonalizable Matrix
A Nondiagonalizable Matrix
Links to the Linear Algebra World
OnLine Linear Algebra Text by Thomas S. Shores
ATLAST Project
Linear Algebra ToolKit
Elements of Abstract and Linear Algebra by Edwin H. Connell
Linear Algebra
Linear Algebra by Jim Hefferon

Linear Algebra Lecture Notes by Keith Matthews
Notes on Linear Algebra by Lee Lady
Electronic Journal of Linear Algebra
Down with Determinants by Sheldon Axler
Linear Algebra Glossary by John Burkhardt
Linear Algebra Notes by Dr. Min Yan (Hong Kong University)
Internet Resources for Linear Algebra - Langara College
Companion Website to Linear Algebra with Applications by Steven Leon
Linear Algebra Calculator
Linear Algebra Print Journals
The College of Wooster Home Page
Last Updated: July 27, 2004

Jim Hartman: [email protected]
Mathematics and Computer

ScienceDept. Home Page
Jim Hartman
Mathematics and Computer Science
Jim
Hartman
Professor of
Mathematics
hartman@wooster.
edu
330-263-2239
The College of
Wooster
Wooster, OH
44692
SUMMER 2005
2005 Summer Institute for AP Calculus

Description
Tentative Schedule
Questions
AP Calculus Reading
May 29- June 9
I can be reached through: jimcowmath
http://www.wooster.edu/math/jhartman.html (1 of 3)2005/03/08 03:50:58 .
Jim Hartman
2005 SPRING SEMESTER COURSES
Math 112 - Section 2
Math 102
Calculus and Analytic Geometry II

Calculus by James Stewart, Brooks/Cole
Exam 1 Review
Exam 1 Solutions
Exam 2 Review
Course Formulas
Basic Statistics
The Basic Practice of Statistics by David S. Moore, Freeman
Course Materials
Senior Independent Study
Antoney Calistes - Differential Geometry

Lauren Gruenebaum - Benford's Law
Rebecca Young - Bootstrap Methods
OFFICE HOURS
Monday
Tuesday
Wednesday
Thursday
Friday
10-12Noon
3-5PM
8-9AM
11-12Noon
3-4PM
4-5PM
By
Appointment
10-12Noon
EDUCATION
B.S. Manchester College 1975
M.S. Michigan State University 1981 (Statistics)
Ph.D. Michigan State University 1981 (Mathematics)
PUBLICATIONS
Frozen in Time, APCentral, July 2004.
Some Thoughts on 2003 Calculus AB Question 6, with Riddle, Larry, APCentral, June
2003.
Problem #726 Solution, College Journal of Mathematics, Volume 34, No. 3, May 2003.
A Terminally Discontinuous Function, College Journal of Mathematics, Volume 27,
No. 3, May 1996.
Problem #E3440 Solution, American Mathematical Monthly, Volume 99, Number 10,
Dec. 1992.
Functions and Maple, CASE NEWSLETTER, Number 12, Jan. 1992.
On a Conjecture of Gohberg and Rodman, Journal of Linear Algebra and Its
Jim Hartman
Applications, 140: 267-278 (1990).

Studying Chaotic Systems Using Microcomputer Simulations and Lyapunov
Exponents, with De Souza-Machado, S., Rollins, R. and Jacobs, D.T., American
Journal of Physics, 58(4),321-329 (1990).
Weighted Shifts with Periodic Weight Sequences, Illinois Journal of Mathematics,
Volume 27, Number 3(1983),436-448.
A Hyponormal Weighted Shift Whose Spectrum is not a Spectral Set, Journal of
Operator Theory, 8(1982), 401-403.
CAREERS IN MATHEMATICS
ACTIVITIES/ORGANIZATIONS
Mathematical Association of America (MAA)
American Mathematical Society
Mennonite Connected Mathematicians
INTERESTS
Linear Algebra
Involutions
Statistics
Operator Theory
3n+1 Problem
A bibliography
HOBBIES
Basketball
Photography
Bicycling
FAMILY
Last updated: 11 February 2005

Jim Hartman [email protected]
The College of Wooster
Campus Tour | Interactive | Wooster Webcam
Popular Wooster Sites
As I have gotten to know

what [Wooster]
accomplishes I can testify
there is no better college
in the country.
Loren Pope, Colleges
That Change Lives
About the College

Academic Programs
Athletics
Directories
Independent Study
Monday, March 7, 2005
Libraries
College meets Walton Foundation challenge, then raises $2

million more
Wooster celebrates International Women's Day with
Peggy Kelsy's Afghan Women's Project
Povertyfighters Collegiate Click Drive Feb. 14-March
31; Register for free with The College of Wooster and click
twice daily
More News
Today's Events
The College of Wooster 2005

All Rights Reserved
http://www.wooster.edu/2005/03/08 03:52:00 .
Search
Contact Wooster
Site Index Site Map
Policies
Course Policies
Grades
Goals
Academic Integrity
Grades
Grade Determination:
Your grade will be based on the total number of points you receive out of a possible total of 1000 points.
The distribution of these points is given below.
2 Hour Exams
(100 points each)
1 Final Exam
(200 points)
Homework
(100 points total)
Writing Assignments
Definition Quizzes
(200 points total)
Proofs
(120 points total)
Research Topic
(60 points)
Matrix Project
(100 points)
Informal Writings
(20 points total)
Missed Exams:
Make-up exams will be given only for valid and verifiable excuses. It is important to notify me before an
exam that you must miss.
http://www.wooster.edu/math/linalg/policies.html (1 of 3)2005/03/08 03:52:14 .
Policies
Final Exam:
Section 1 -Friday, December 12, 2:00 PM
Section 2 - Thursday, December 11, 7:00 PM
Math 211 Home Page
Course Goals
To learn about matrices and their properties.

To learn about vector spaces and inner product spaces.
To see different examples of these spaces.
To learn about linear transformations on these spaces.
To learn about applications of linear algebra.
To learn how to construct proofs of known theorems.
To improve mathematics writing with attention to the writing process
Math 211 Home Page
Academic Integrity
I encourage students to exchange ideas and discuss problems. However, for homework to be turned in,
it will be considered plagarism if a student copies the work of another. On exams or quizzes the giving
or receiving of aid is not permitted. Any violation of the Code of Academic Integrity should be reported
to the instructor who will take appropriate disciplinary action and/or inform the Judicial Board of the
case. In either case, the Dean's office will be notified of the violation.
Code of Academic Integrity
Policies
Math 211 Home Page
Lecture Schedule
TENTATIVE LECTURE SCHEDULE

DATE
TOPIC
August 30
1.1
September 1
1.2
2 Problems
3
2.1
25
27
28
29
TOPIC
Quiz 6
5.2
Problems
5.3
November 1
3
4
5
Quiz 7
5.4
Problems
6.1
13 Quiz 2
15
2.5
16 Problems
17
3.1
8
10
11
12
Quiz 8
6.2
Review
EXAM 2
20 Quiz 3
22
3.2
23 Problems
24
3.3
15
17
18
19
Quiz 9
6.3
Problems
6.4
27
3.4
29
3.5
30 Review
October 1 EXAM 1
22
24
25
26
Quiz 10
Break
Break
Break
29
December 1
2
6.5
6.6
LAB #3
6
8
9
10
4
6
7
Quiz 1
2.2
Lab#1
2.3
Quiz 4
4.1
LAB #2
DATE
http://www.wooster.edu/math/linalg/lecture.html (1 of 2)2005/03/08 03:52:27 .
Lecture Schedule
4.2
11 Quiz 5
13
4.3
14 Problems
15
4.4
FALL
BREAK
20
4.5
21 Problems
22
5.1
18
7.1
6
8
9
10
7.2
7.3
Problems
REVIEW
FINALS WEEK
13
14
15
16
Return to Math 211 Home Page
http://www.wooster.edu/math/linalg/lecture.html (2 of 2)2005/03/08 03:52:27 .
FINAL - 9am
FINAL - 7pm
History
A Brief History of Linear Algebra

There would be many things to say about this theory of matrices
which should, it seems to me, precede the theory of determinants.
Arthur Cayley, 1855
Determinants Used Before Matrices
1693
1750
Leibniz
Cramer
solving systems of equations
Implicit Use of Matrices
Late 18th century

Lagrange
bilinear forms for the optimization of a real valued function of 2 or more variables
Gaussian Elimination
1800
Gauss
Method known by Chinese for 3x3 in 3rd century BC
Vector Algebra
1844
Grassmann
http://www.wooster.edu/math/linalg/history.html (1 of 3)2005/03/08 03:52:33 .
History
Matrix Algebra
1848
J. J. Sylvester
introduced the term matrix which is the Latin word for womb (determinants emanate from
a matrix)
define the nullity of a matrix in 1884
1855
Arthur Cayley
definition of matrix multiplication motivated by composite transformations, also
introduced inverses
1878
George Frobenius
introduced rank and proved the Cayley-Hamilton Theorem
Vector Space
1888
Peano
modern definition of vector space
Further Developments
1942
J. Willard Gibbs
further development of ideas by this mathematical physicist
1942
John von Neumann
condition number
1948
Alan Turing
LU decomposition
1958
J. H. Wilkinson
QR factorization
Linear Algebra in the Curriculum
1941
Birkhoff and MacLane

appearance in the basic graduate text Modern Algebra
1959
Kemeny, Snell, Thompson, & Mirkel
appearance in the undergraduate text Finite Mathematical Structures
1960
Widespread adoption into lower division mathematics curriculum
History
1965
CUPM Recommendations
suggest linear algebra be a course in the undergraduate curriculum and content for the
course
1990
New CUPM Guidelines look at the focus of the linear algebra course
1990's Experimentation with technology and content
Math 211 Home Page
Writing
WRITING ASSIGNMENTS
DEFINITION QUIZZES
PROOFS
MATRIX PROJECT
RESEARCH TOPIC
INFORMAL WRITINGS
Math 211 Home Page
DEFINITION QUIZZES
VALUE: 200 points total
Quizzes will be given almost every Monday of the semester during the first 10-15 minutes of class.
These quizzes will only ask you to state definitions of terms used. Those definitions will come from the
material studied in the previous class periods that have not been previously quizzed. There are more
definitions than those simply labeled with the word "definition" in the book. Terms not appearing in the
book, but given in class, might also be tested.
TO TOP
PROOFS
DUE DATES: To be determined
VALUE:
120 points total (30 points each)
You are asked to submit proofs of the following theorems. These proofs will, after revision, be clear,
complete, and accurate. These are theorems that might be referred to in class, but will not be proved.
The proofs should be written with the audience being your fellow classmates.
Theorem 1:
For vectors u and v in n, ||u + v|| ||u|| + ||v||.
http://www.wooster.edu/math/linalg/writeassign.html (1 of 3)2005/03/08 03:52:45 .
Writing
Theorem 2:
The vectors v1,v2, . . . ,vn are linearly independent. If w is not in span(v1,v2, . . . ,vn) then w,v1,v2, . . . ,
vn are linearly independent.
Theorem 3:
Theorem 4:
TO TOP
MATRIX PROJECT
First Draft Due: November 2004
Final Draft Due: December 10, 2004
VALUE: 100 points
You will be given three matrices. Your assignment is to describe anything about these matrices that you
can. Some of the ideas that you should explore as you work with these matrices are:
1) Examine any interesting arithmetic properties
2) Examine what happens when I "combine" this matrix with others
3) Examine whether this matrix fits into a more general class of matrices. If so,
find the properties of that class. Determine whether that class is a subspace
of all matrices of the appropriate "size."
4) Find characteristics about the matrix that we have studied in class, such as
determinant, row space, column space, null space, rank, eigenvalues and
eigenvectors.
An example of the kind of work I would like to see can be obtained from me personally, or retrieved
from the course website. Your paper should be written as if it was going to be read by your fellow
students. Thus you should assume they have the basic knowledge from this course, but nothing beyond.
One goal of this assignment is to show that you understand the concepts that have been covered in this
course. A second goal is for you to begin to explore properties and structures that you may not have
encountered before and to begin to ask your own questions about what you are encountering. This is a
creative exercise and you may do anything (legal) to these matrices that you desire. Use your
imagination.
Your final paper should be completed in Word or Maple. It should be double-spaced and be of a
length that is sufficient to describe the properties of each matrix. If you get help from other sources, they
Writing
need to be cited and referenced. You are free to determine the organization of your paper, but should
express your ideas in clear and concise ways with flawless grammar and spelling. This is both a writing
assignment and a mathematics assignment. You are writing about mathematics that you are exploring. I
will read and comment on the first draft. You will then be able to make revisions based upon my
comments. The grade for the paper will be based upon the final draft, although I will give you a
preliminary grade based upon your first draft to give you an idea of where you stand.
The paper will be graded based upon a primary trait analysis which will be provided later.
TO TOP
RESEARCH TOPIC
DUE DATE:
VALUE:
First part of the semester

60 points
During the first part of the semester, I will ask you to research some topic in linear algebra. This topic
will either be chosen from a list or be an approved topic of your own choice. You will need to provide a
bibliography with this paper along with inline citations.
TO TOP
INFORMAL WRITINGS
DUE DATE:
VALUE:
December 10, 2004

20 points
Throughout the semester, I will ask you to complete writings both in and out of class. Some of these
may be freewrites, while others I will ask you to construct more carefully or on a particular subject.
Some will be submitted in class; others will be submitted by email. The points given here will be
primarily for the completion of and effort given to these writing tasks rather than the quality of the
writing.
TO TOP
Matrix Calculator
Search:
Angelfire
Build an Online Photo Album
Web by
Go!
Try Blogging for FREE
Matrix Calculator - NEW VERSION AVAILABLE!

Mcalcdos is a free command line linear algebra calculator capable of matrix, vector and polynomial operations in real and
complex fields.
Download
Download mcalcdos.exe v0.3 for free!!
A newly updated version of mcalcdos.exe is now available! Please click above to download the latest version, which includes an
up-to-date help file.
Click here for updates made to the current release.
To use, simply input mathematical expressions in the same manner as written to return the result. For a list of available
functions, type help. To ease input, multiplication is implied between adjacent operands.
e^(2 pi i) + 0.5 cos (2 pi/3)
http://www.angelfire.com/space/matrixcalc/ (1 of 2)2005/03/08 03:54:11 .
Matrix Calculator
Variables are assigned using = as in the following example:

a = (1 + sqrt 5) / 2
Once assigned, variables can be used in subsequent expressions. The variable ans is automatically updated to the result of the
previous expression. Other reserved names include pi, i, e, X (which is used in polynomial expressions) and all the function
names. Any other variable names are permitted.
varname = 14 e ^ 0.7
newvar = 3.7! + varname
Input matrices with [], vectors with {} and polynomials with X
matrix_A = [4, 7][2, 3]
vector_b = {1, -2}
inverse a * b
eval (X^3 - 7X + 1, -3)
Eventually, this program will be given a windows GUI, but until then, feel free to use and distribute the text based version.
Please contact me with any questions, comments or for (very) limited support.
Email: [email protected]
OR
[email protected]
http://www.angelfire.com/space/matrixcalc/ (2 of 2)2005/03/08 03:54:11 .
MATH 211
TEST I
Name
SOLUTIONS
Fall 2004
PLEASE SHOW ALL OF YOUR WORK!!
1.(15 PTS) Consider the system of equations given by Ax=b, and let C = [A | b] be the
augmented matrix for that system. If G=[M | d] is the reduced, row-echelon form
of C:
a) What will you see in G if the system is inconsistent?
b) What will you see in G if the system has exactly one solution?
c) What will you see in G if this system has infinitely many solutions?
a)
b)
c)
There will be a leading 1 in the last column of G, i.e. in d.

There will be a leading 1 in every column of M and not in d.
There will be a column of M without a leading 1 and d will not have a
leading 1.
2.(10 PTS) Suppose we have the system of equations given by Ax=b. Let x1 be a particular
solution to this equation. Show that if x2 is any other solution to this system then
x2 - x1 is a solution to the homogeneous system Ax=0.
A(x2 x1) = Ax2 A x1 = b b = 0.
3.(10 PTS) Through two examples illustrate how matrix multiplication does not satisfy two
distinct properties that real number multiplication does.
Consider the following two examples.
"1 1% " 2 2 %
"0 0%
$ '$
' = $
'
#1 1& #-2 -2&
#0 0&
and
# 2 2 & #1 1&
#4 4&
%$"2 "2(' %$1 1(' = %$-4 -4('
These two equations show two things. First, matrix multiplication is not
commutative. Second, the product of two nonzero matrices can be zero.
!
!
#2
4.(10 PTS) M = %% 2
%$ 3
a)
!
5
2
3
#1
6 "5 0 &
(
%
"6 "5 "3( has rref (M) = % 0
%$ 0
"9 "2 1 ('
0
1
0
"7
4
0
0
0
1
0&
(
1 (.
1 ('
Express any columns of M, that are possible, as linear combinations of other

columns of M.
!
Let m1, m2, m3, m4, m5 be the columns of M. Then m3 = -7m1 + 4m2
and m5 = m2 + m4.
b)
Find a spanning set for the homogeneous system of linear equations

given by Mx = 0.
The solutions to Mx = 0 are given by x1 = 7x3, x2 = -4x3 x5 and x4 = -x5.
This leads to a solution vector of
# 7x 3 &
#7&
#0&
%
(
% (
% (
%"4 x 3 " x 5 (
%"4(
%"1(
% x 3 ( = x 3 % 1 ( + x 5 % 0 ( . Thus a spanning set is
%
(
% (
% (
% "x 5 (
%0(
%"1(
%$ x 5 ('
%$ 0 ('
%$ 1 ('
)# 7 & # 0 &+% ( % (+
++%"4( %"1(++
*% 1 (,% 0 (. .
+% 0 ( %"1(+
+% ( % (+
+,%$ 0 (' %$ 1 ('+/
5.(10 PTS) Suppose u, v, and w are nonzero orthogonal vectors.

2
Show that ||u + v + w||2 = ||u||2 + ||v||2 + ||w||
! a)
! .
||u + v + w||2 = (u + v + w)( u + v + w)
= uu + uv + uw + vu + vv + vw + wu + wv + ww
= ||u||2 + 0 + 0 + 0 + ||v||2 + 0 + 0 + 0 + ||w||2
= ||u||2 + ||v||2 + ||w||2
b)
Show that u, v, w are linearly independent.

If u + v + w = 0 then u(u + v + w) = u0 = 0 so
uu + uv + uw = 0 or ||u||2 + (0) + (0) = 0 or ||u||2 = 0. Since
||u||2 0 we must have = 0. Similarly be considering
v(u + v + w) = v0 = 0 and w(u + v + w) = w0 = 0 we also get
= = 0. Since we have = = = 0, the vectors u, v, and w are linearly
independent.
6.(10 PTS) Suppose u, v, and w are linearly independent vectors. Show that
u + v + w, v + w, v w are linearly independent.
If ( u + v + w) + (v + w) + (v - w)w = 0 then u + (+)v + (-)w = 0. Since
u, v, and w are linearly independent we must have = 0, + = 0 and - = 0. The
last two equations imply that = = 0. Combining this with = 0 says the vectors
are linearly independent.
7.(10 PTS) A vector y in m is said to be the in the range of the matrix A if there is a
vector x in n such that y = Ax.
a)
Show that if y1 and y2 are in the range of A then so is y1 + y2.
y1 = Ax1 and y2 = Ax2 so that y1 + y2 = Ax1 + Ax2 = A(x1 + x2) so y1+y2 is in
the range of A.
b)
Show that if is a scalar and y is in the range of A then y is in the range of

A.
y = Ax = A(x) so y is in the range of A also.
MATH 211
TEST I
TAKE HOME PROBLEMS
Name
Solutions for this part are attached.
Fall 2004
Note: If you use MAPLE on these problems, please provide me with a printed copy of all the
work you did. You can label that printout or add to it to indicate answers to questions.
DUE DATE: Wednesday, 6 October 2004 - 4:00 PM (EDT)
# 8 & #1& #"5&
% (% (% (
% 4 ( %0( % 8 (
8.( 10 PTS) Consider the list of vectors %"5(,%0(,% 5 (.
% (% (% (
%"5( %0( %"1(
%$ 3 (' %$0(' %$ 0 ('
a)
b)
Show that the 3 vectors above are linearly independent.

Find two more
! vectors in 5 so that the span of the 5 vectors is 5.
Indicate how you know that your 5 vectors span 5.
9.(15 PTS) An mxn matrix A is said to have a left inverse if there is an nxm matrix B so that
BA = Im.
a)
#8
%
%4
Find a left inverse for the matrix M = %"5
%
%"5
%$ 3
1
0
0
0
0
"5&
(
8(
5 (.
(
"1(
0 ('
b)
Does M have only one left inverse? Don't just answer yes or no to this
question. If there is more than one, find a second one. If there is exactly
one, prove it. Just how many
left inverses, does M have.
!
c)
What conditions must be placed upon an mxn matrix M to guarantee that it

has a left inverse?
d)
A matrix A is said to have a right inverse if there is a matrix B so that AB=I.

Show that the matrix M above cannot have a right inverse.
Linear Algebra
Exam 1 Maple Solutions
> with(LinearAlgebra):
Problem 8
> v1:=<8,4,-5,-5,3>:v2:=<1,0,0,0,0>:v3:=<-5,8,5,-1,0>:
Part a)
> M:=<v1|v2|v3>;
M :=
-5
-5
-5
-1
> ReducedRowEchelonForm(M);
Since there is a leading 1 in every column of rref(M), the columns of M are linearly independent.
Part b)
> N:=<M|IdentityMatrix(5)>;
8
N := -5
-5
3
> ReducedRowEchelonForm(N);
-5
-1
-5
-1
-11
-5
3
12
10
This matrix has leading 1's in columns 1,2,3,5,6 and indicates that the corresponding 5 columns of N are
linearly independent and should span R^5. One can see this from the following three Maple commands.
> v4:=SubMatrix(N,1..5,5):v5:=SubMatrix(N,1..5,6):
> V:=<v1|v2|v3|v4|v5|<x,y,z,w,e>>;
1
8
4
0
V := -5
0
-5
0
3
0
> ReducedRowEchelonForm(V);
0
1
0
1
0
0
0
0
0
0
-5
-1
1
5
x - 5 w - 11 e
5
- e - w
3
8 w + y + 12 e
10 e + z + 5 w
1
e
3
This last matrix shows that any vector <x,y,z,w,e> in R can be written as a linear combination of the
first 5 columns of V. Thus the set
> {v1,v2,v3,v4,v5};
-5
-5
-5
-1
is a basis for R .
Problem 9
Part a)
> C:=<v1|v2|v3|v4|v5>;
C :=
-5
-5
-5
-1
> X:=C^(-1);
X :=
-5
-1
-5
-1
> A:=SubMatrix(X,1..3,1..5);
A := 1
> A.M;
-11
-5
3
12
10
1
-11
-5
0
1
0
Thus the matrix A above is a left inverse for M.
Part b)
There are infinitely many left inverses for M. Consider the following computations.
> A1:=<<x1,x2,x3>|<x4,x5,x6>|<x7,x8,x9>|<x10,x11,x12>|<x13,x14,x15>>;
x4
x7
x10
x13
x1
A1 := x2
x5
x8
x11
x14
x6
x9
x12
x15
x3
> H:=A1.M;
8 x1 + 4 x4 - 5 x7 - 5 x10 + 3 x13
H := 8 x2 + 4 x5 - 5 x8 - 5 x11 + 3 x14
8 x3 + 4 x6 - 5 x9 - 5 x12 + 3 x15
x1
x2
x3
-5 x1 + 8 x4 + 5 x7 - x10
-5 x2 + 8 x5 + 5 x8 - x11
-5 x3 + 8 x6 + 5 x9 - x12
> sol:=solve({H[1,1]=1,
H[1,2]=0,H[1,3]=0,H[2,1]=0,H[2,2]=1,H[2,3]=0,H[3,1]=0,H[3,2]=0,H[3,3]=1});
5
sol := x15 = 12 x6 + 10 x9 - , x10 = 8 x4 + 5 x7, x1 = 0, x2 = 1, x3 = 0, x11 = -5 + 8 x5 + 5 x8,
3
1
x12 = -1 + 8 x6 + 5 x9, x14 = -11 + 12 x5 + 10 x8, x13 = + 12 x4 + 10 x7, x4 = x4, x5 = x5,
3
x6 = x6, x7 = x7, x8 = x8, x9 = x9
> assign(sol);
> A1;A1.M;
x4
x7
8 x4 + 5 x7
x5
x8
-5 + 8 x5 + 5 x8
x6
x9
-1 + 8 x6 + 5 x9
1
0
1
0
1
+ 12 x4 + 10 x7
3
-11 + 12 x5 + 10 x8
5
12 x6 + 10 x9 -
3
Every matrix of the form of A1 above is a left inverse for M as indicated by the computation above.
There are infinitely many since the free variables x4,x5,x6,x7,x8,x9 can be chosen to be any real
numbers. If we let them all be zero then
> x4:=0:x5:=0:x6:=0:x7:=0:x8:=0:x9:=0:A1;
0
0
0
0
1
0
0
-5
0
0
-1
0
-11
-5
the matrix above is a particular second left inverse for M.
Part c)
The columns of M must be linearly independent in order for M to have a left inverse. This can be seen in
a couple of ways. One way uses the idea I used to construct the first left inverse of M. If the columns of
M are linearly independent, one can add two other columns to get an invertible matrix. This matrix will
yield a left inverse by taking the upper 3x5 submatrix of the inverse. If the columns are not linearly
independent this process is not possible. moreover if A is a left inverse and Mx=0 then x = Ix = AMx =
A0 = 0 and hence the columns of M are linearly independent. These two parts together show M has a left
inverse iff the columns of M are linearly independent.
Part d)
If M had a right inverse then every vector in R5 could be written as a linear combination of the columns
of M. But we already know the vector <0,1,0,0,0> cannot be written as a linear combination of the
columns of M. This was seen in part a) but can also been seen in:
> N:=<M|<0,1,0,0,0>>;
N :=
-5
-5
-5
-1
> ReducedRowEchelonForm(N);
The leading 1 in the last column verifies the statement above.
Review 1
Math 211 Linear Algebra

Exam 1
Fundamental Ideas
A. Systems of Equations
1. Augmented Matrix
2. RREF
3. Ax = b
B. Vectors
1. Addition and Scalar Multiplication
2. Dot Product , Norm, and Orthogonality
3. Orthogonal Projection
C. Matrices
1.
2.
3.
4.
5.
6.
Addition, Scalar Multiplication, and Multiplication

Inverses
Transpose
Elementary Matrices
Rank
RREF
D. Span and Linear Combinations

E. Linear Independence
http://www.wooster.edu/math/linalg/review1.html2005/03/08 03:57:36 .
1
MATH 211
EXAM 2
Name
PLEASE SHOW ALL OF YOUR WORK!!
1.(10 PTS)
SOLUTION
2.(18 PTS)
Fall 2004
+# x " 2y &
/
-%
(
-% 2y " x (
4
Find a basis for the subspace of given by ,
: x, y, x ) *0 . What is the
-% x " 2y + z(
(
-.%$
-1
z
'
dimension of this subspace?
!
# x " 2y & # 1 & #"2& #0&
# 1 & #0&
%
( % ( % ( %(
% ( %(
% 2y " x ( = x%"1( + y% 2 ( + z%0( = ( x " 2y )%"1( + z%0(. Thus a basis for the subspace is
%x " 2y + z( % 1 ( %"2( %1(
% 1 ( %1(
%
( % ( % ( %(
% ( %(
z
$
' $ 0 ' $ 0 ' $1'
$ 0 ' $1'
)# 1 & #0&+% ( % (+
+%"1( %0(+
* , . and its dimension is 2.
+% 1 ( %1(+
+,%$ 0 (' %$1('+/
# 8
%
% 4
For A = % "5
%
% "5
%$ 3
"5
8
5
"1
0
31
"16
"25
"7
6
"2
20
5
"7
3
#1
3 &
(
%
"4 (
%0
"5 (, rref (A) = % 0
(
%
"2 (
%0
%$ 0
"1 ('
0
1
0
0
0
2
"3
0
0
0
1
2
0
0
0
0&
(
0(
1 (.
(
0(
0 ('
a) Give a basis for col(A).
!
SOLUTION
!
)# 8 & #"5& # 3 &+% ( % ( % (+
++% 4 ( % 8 ( %"4(++
*%"5(,% 5 (,%"5(. is a basis for col(A).
+%"5( %"1( %"2(+
+% ( % ( % (+
+,%$ 3 (' %$ 0 (' %$"1('+/
b) Give a basis for row(A) .

!
SOLUTION
{[1
0 2 1 0], [0 1 "3 2 0], [0 0 0 0 1]} is a basis for the row space.
2
c) Give a basis for null(A).
SOLUTION
" x1 %
$ '
$x2 '
A vector $ x 3 ' will be in the nullspace if and only if x1 + 2x3 + x4 = 0, x2 3x3 + 2x4 = 0,
$ '
$x 4 '
$# x 5 '&
)#"2& #"1&" x1 % "(2x 3 ( x 4 %
"(2%
"(1%
+% ( % (+
$ ' $
'
$ '
$ '
x
3x
(
2x
3
(2
2
3
4
++% 3 ( %"2(++
$ ' $
'
$ '
$ '
!and x5 = 0. This implies $ x 3 ' = $ x 3 ' = x 3 $ 1 ' + x 4 $ 0 ' . Thus *% 1 (,% 0 (. is a basis
$ ' $
'
$ '
$ '
+% 0 ( % 1 (+
$x 4 ' $ x 4
'
$0'
$1'
+% ( % (+
+,%$ 0 (' %$ 0 ('+/
$# x 5 '& $#
'&
$# 0 '&
$# 0 '&
0
for null(A).
c) What is rank(A)?
!
SOLUTION
rank(A) = 3
d) What is nullity(A)?
SOLUTION
nullity(A) = 3
f) Is A invertible? Why or why not?
SOLUTION
A is not invertible because rank(A) = 3 < 5. To be invertible, its rank would have to be 5.
3.(10 PTS)
#
%
%
The columns of B = %
%
%
$
1
1
1
1
2
2
2
2
1
1 &
2
2
2(
" 1 2 1 2 " 1 2(
form an orthonormal basis for 4.
1
1 "1 (
"
2
2
2(
1
1
1
(
" 2 " 2
2'
1
"1%
$ '
2
a) Write the vector $ ' as a linear combination of the columns of B.
$3'
$ '
#4&
SOLUTION
(
" %+" % (
"
%+"
% (
"
%+"
% (
"
%+"
%
"1% *"!
1% $ 1 2'-$ 1 2' *"1% $ 1 2 '-$ 1 2 ' *"1% $ 1 2 '-$ 1 2 ' *"1% $ 1 2 '-$ 1 2 '
$ ' *$ ' 1 - 1
*$ ' . 1 - . 1
*$ ' 1 - 1
*$ ' . 1 - . 1
$2' = *$2' $ 2'-$ 2' + *$2' $ 2'-$ 2' + *$2' $ 2 '-$ 2 ' + *$2' $ 2'-$ 2'
$3' *$ 3' $ 1 '-$ 1 ' *$3' $ 1 '-$ 1 ' *$ 3' $. 1 '-$. 1 ' *$3' $. 1 '-$. 1 '
$ ' *$ ' $ 2'-$ 2' *$ ' $ 2 '-$ 2 ' *$ ' $ 2'-$ 2' *$ ' $ 2'-$ 2'
#4& #4& $# 1 '& $# 1 '& #4& $#. 1 '& $#. 1 '& #4& $#. 1 '& $#. 1 '& #4& $# 1 '& $# 1 '&
)
2, 2 )
2,
2 )
2,
2 )
2 , 2
"1 %
"1 %
"1 %
"1 %
$ 2'
$ 2'
$ 2'
$ 2'
$ 1 2'
$. 1 2'
$ 12 '
$. 1 2'
= (5)$ ' + (.1)$
+ (.2)$
+ (0)$
1
1 '
1 '
1 '
$ 2'
$ 2'
$. 2'
$. 2'
$1 '
$. 1 '
$. 1 '
$1 '
# 2&
# 2&
# 2&
# 2&
b) What is B-1?
!
SOLUTION
#
%
%
B"1 = BT = %
%
%
$
1
1
1
1
2
2
2
2
1
1 &
2
2
2(
" 1 2 1 2 " 1 2(
(= B
1
"1 "1 (
2
2
2
" 1 2 " 1 2 1 2 ('
1
#"1 9 &
4.(12 PTS) ! Consider the matrix B = %
(.
$ 6 "4'
a) Find eigenvalues and corresponding eigenvectors for B.
SOLUTION
!
$" + 1 #9 '
2
"I # B = &
) and det(I-B) = (+1)(+4) 54 = + 5 - 50 = ( + 10)( - 5).
#6
"
+
4
%
(
Thus B has two eigenvalues -10 and 5.
)# 6 "9&,
)#3&,
For = 5 we need null(5I B) = null+%
(. = span+% (. and for = -10 we need
*$"6 9 '*$2')#"9 "9&,
)#"1&,
null(-10I B) = null+%
(. = span+% (. . The vectors that span those nullspaces are
*$"6 "6'*$ 1 'eigenvectors corresponding
! to the eigenvalues.
!
4
b) Describe the eigenspaces for each of the eigenvalues.
SOLUTION
("3%+
)#"1&,
The two eigenspaces are E 5 = span*$ '- and E "10 = span+% (. .
)#2&,
*$ 1 '-
c) Give the algebraic and geometric multiplicities of each of the eigenvalues.

!
SOLUTION
The algebraic and geometric multiplicity of each eigenvalue is 1.
5.(10 PTS)
6.(10 PTS)
Let M be a 3x5 matrix with rank(M)=3. Fill in the following blanks.

a)
The columns of M are linearly
b)
The rows of M are linearly
c)
nullity(M) =
d)
rank(MT) =
e)
nullity(MT) =
dependent
independent
. (independent or dependent)
. (independent or dependent)
"2y . x%
("x%+ $
'
Let T:2 3 be given by T*$ '- = $ x . 3y'.
)#y&, $
# y 2 '&
a) Show that T is a linear transformation.
SOLUTION
!
"2y . x% " .1
("x%+ $
' $
T*$ '- = $ x . 3y' = $ 1
)#y&, $
# y 2 '& $# 0
2%
' " x%
.3 ' $ ' . This implies T is a matrix transformation and hence is
# y&
2 '&
a linear transformation.
b) Find the matrix of T with respect to the standard basis.
SOLUTION
# "1
%
The matrix of T is % 1
%0
$
2&
(
"3 ( .
2 ('
5
7,(5 PTS)
Let A and B be matrices with C = AB. Show that null(B) null(C). What does this say
about rank(C) in comparison to rank(B)?
SOLUTION
If x null(B) then Bx = 0. So Cx = ABx = A0 or (AB)x = 0. Thus x null(C). This says
null(B) null(C).
Since nullity(B) nullity(C) and B and C have the same number of columns,
we must have rank(C) rank(B) since B and C have the same number of columns and
rank(M) + nullity(M) = the number of columns of M for any matrix M.
MATH 211
EXAM 2
TAKE HOME PROBLEMS
Name
SOLUTIONS
Note:
Fall 2004
If you use MAPLE on these problems, please provide me with a printed copy of all the
work you did. You can label that printout or add to it to indicate answers to questions.
DUE DATE: Wednesday, 13 November 2004 - 4:00 PM (EST)

THE SOLUTIONS TO THIS PART ARE FOUND BELOW!
# 9 "17 1
15 &
%
(
"15
7
17
"1
(.
8.(16 PTS) Consider the matrix A = %
% 1
15
9 "17(
%
(
"1 "15 7 '
$ 17
9.(9 PTS)
a)
Find!all eigenvalues for A.
b)
Determine the algebraic multiplicity of each of the eigenvalues for A.
c)
Find bases for each of the eigenspaces for A.
d)
Determine the geometric multiplicity of each of the eigenvalues for A.
# 5 "8
%
%"7 4
Let B = % 0 0
%
%1 8
%$ 9 0
0
0
4
0
0
1 8&
(
9 0(
0 0 ( . Show how to diagonalize B.
(
5 "8(
"7 4 ('
EXAM 2
TAKE HOME SOLUTIONS
FALL 2004
Problem 8
> A:=<<9,-15,1,17>|<-17,7,15,-1>|<1,17,9,-15>|<15,-1,-17,7>>;
-17
1
15
9
-15
7
17
-1
A :=
1
15
9
-17
-1
-15
7
17
Part a)
> ei:=Eigenvectors(A,output=list);
ei :=
-24, 1,
-1
-1
, 40, 1,
-1
-1
, 8, 2,
A has 3 eigenvalues. They are -24, 40, and 8.
Part b)
The algebraic multiplicity of -24 is 1. The algebraic multiplicity of 40 is 1. The algebraic multiplicity of 8
is 2.
Part c)
A basis for E-24 is:
> ei[1][3];
-1
-1
A basis for E40 is:

> ei[2][3];
-1
-1
A basis for E8 is:

> ei[3][3];
Part d)
The geometric multiplicity of each of the three eigenvalues is 1.
Problem 9
> B:=<<5,-7,0,1,9>|<-8,4,0,8,0>|<0,0,4,0,0>|<1,9,0,5,-7>|<8,0,0,-8,4>>;
-8
0
1
8
5
-7
4
0
9
0
B := 0
0
4
0
0
8
0
5
-8
9
0
0
-7
4
> eib:=Eigenvectors(B,output=list);

-1
1

-1
1

eib := -12, 1, 0 , 6, 1, 0

1
1

1
1

, 4, 2,
, 20, 1,
-1
-1
> v1:=eib[1][3][1];v2:=eib[2][3][1];v3:=eib[3][3][1];v4:=eib[3][3][2];v5:
=eib[4][3][1];
-1

-1

v1 := 0

1

1
> P:=<v1|v2|v3|v4|v5>;
v2 :=
v3 :=
v4 :=
v5 :=
-1
-1
P :=
-1
-1
-1
-1
> (P^(-1)).B.P;
-12
20
Review 2
Math 211 Linear Algebra

Exam 2
Fundamental Ideas
A. Vectors
1.
2.
3.
4.
Subspace
Basis
Dimension
Coordinate Vectors
B. Matrices
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Row Space
Column Space
Null Space
Rank
Nullity
Fundamental Theorem of Invertibility
Eigenvalues
Eigenvectors
Similarity
Diagonalization
D. Linear Transformations
1.
2.
3.
4.
Definition
Matrix Transformation
Matrix of a Linear Transformation
Inverse Transformation
E. Determinants
1. Definition
2. Properties
http://www.wooster.edu/math/linalg/review2.html2005/03/08 04:00:32 .
Lecture Notes
Lecture Notes
Math 211
Linear Algebra
(Based upon David Poole's Linear Algebra: A Modern Introduction)
Chapter 1 Section 1
Chapter 2 Lecture Notes
Chapter 3 Section 2
Incomplete
Incomplete
http://www.wooster.edu/math/linalg/lectnotes.html2005/03/08 04:00:34 .
Incomplete
commandlist.html
Maple Commands for Linear Algebra
Row and Column Operation

Creation of Matrices
Creation of Vectors
Matrix and Vector Arithmetic
Other Operations on Matrices
Download Copy of Command Summary
Row and Column Operations

Elementary Row Operations
RowOperation(A,[m,n])
RowOperation(A,n,c)
RowOperation(A,[m,n],c)
Elementary Column Operations

ColumnOperation(A,[m,n])
ColumnOperation(A,n,c)
ColumnOperation(A,[m,n],c)
Row(A,i..k)
DeleteRow(A,i..k)
Column(A,i..k)
DeleteColumn(A,i..k)
Return to Top of Page
Creation of Matrices
<A1|A2| . . . |An>
Matrix(m,n,f)
Matrix(m,n,symbol=a)
RandomMatrix(m,n,generator=i..j)
IdentityMatrix(n)
<v1|v2| . . . |vn>
Matrix([row1,row2, . . . ,rowm])
SubMatrix(A,rrnge,crnge)
http://www.wooster.edu/math/linalg/command.html (1 of 3)2005/03/08 04:04:38 .
commandlist.html
Creation of Vectors
Vector(n,f)
Vector(n,symbol=v)
<a1,a2, . . . ,an>
SubVector(A,r,crange)
Matrix and Vector Arithmetic

MatrixInverse(A) or A^(-1)
Transpose(A)
Add(A,B) or A+B
DotProduct(u,v)
Multiply(A,B) or A.B
ScalarMultiply(A,expr) or expr*A
MatrixPower(A,n) or A^n
MatrixExponential(A,t)
Other Operations on Matrices

Adjoint(A)
Eigenvalues(A)
Trace(A)
NullSpace(A)
Norm(A,normname)
Determinant(A)
Eigenvectors(A)
Rank(A)
Dimension(A)
Map(f,A)
commandlist.html
MATRIX PROJECT PARTIAL EXAMPLE

Let us try to learn everything we can about the matrix A below.
One thing we can easily note by looking at the matrix is that it is
symmetric. The other thing we might not is that the matrix has crossdiagonals that are constant . To be clear about this we note that
=
=
=
and that
=
=
=
=
. It is unclear
what the number
does for this matrix. We will hopefully find out as
we go along. One other thing we might note is that if we multiply the

first column by
we will get the second column. If we multiply the first
column by
and then by
we will get the third and fourth columns
respectively. Thus the second, third, and fourth columns are scalar
multiples of the first. This says that the rank(A) =1 since the first
column will span Col(A). Note also that this observation about the
columns is true for the rows also since the matrix is symmetric.
I will start by entering this matrix into Maple and doing various
computations. As I do the computations I will make comments
concerning what I am trying to learn. The first thing we might observe is
the rather innovative way that I construct the matrix A. If I let
v:=matrix([[1],[1/2],[1/4],[1/8]]);
then the matrix A =
. We will examine later in this exposition
Page 1
what this way of constructing a matrix will lead to.

> A:=evalm((64/85)*v&*transpose(v));
We first find the determinant of the matrix A. We note that since A has
rank 1 it will not be invertible and hence its determinant should be 0.
> det(A);
Since A is not invertible it will have a nontrivial nullspace. The nullspace is

spanned by the three vectors given below. We note that this also verifies
that the rank is 1, since
rank + nullity = number of columns = 4.
> nullspace(A);
>
rank(A);
We now will find both the charateristic polynomial of A and also all
eigenvalues along with their associated eigenvectors.
> M:= t -> evalm(t*id(4)-A);
>
M(t);
>
p:= t -> det(M(t));
Because of the definitions in Maple above we have that the characteristic

polynomial of A is given by:
> p(t);
Page 2
To find the eigenvalues of A we set p(t) = 0 and solve.

> solve(p(t)=0);
We see from this that 0 is an eigenvalue. We already knew this since
N(A) was nontrivial. Any eigenvector associated with the eigenvalue 0 is
an element of N(A). In fact any nonzero element of N(A) is an
eigenvector with eigenvalue = 0. Now we discover the eigenvectors for
the eigenvalue 1.
> M(1);
>
nullspace(M(1));
>
evalm(A-A^2);
This command tells us that A = A*A so in fact any postive integer power
of A will be A again:
(A*A*A = A*(A*A) = A*A = A etc.).
> rref(A);
This is the reduced, row echelon form of A which we should have guessed
since rank(A) = 1, and hence the first row of A would serve as a basis for
the rowspace(A).
> adj(A);
Page 3
We could have guessed this also since all rows of A are scalar multiples of
the first one. Hence any 3x3 minor submatrix of A will have their rows a
scalar multiple of the first and as a result will not be invertible. Thus any
cofactor ( the determinant of a minor submatrix) will be 0.
We examine several of the norms for a matrix. For example, the spectral
norm is
> norm(A,2);
The 1 norm is the maximum of the sum of the absolute values of the
column entries. This turns out to be:
> norm(A,1);
Since the matrix is symmetric the 1 norm of A and the norm of A
should be the same. The following shows that they are.
> norm(A,infinity);
The Froebenius norm of A is the square root of the sum of the squares of
the entries. For that we get:
> norm(A,frobenius);
>
trace(A);
Now lets examine what happens when we construct any matrix the same
way I did for A in Maple. So let w be the 4x1 matrix with arbitrary entries
a,b,c, and d. So
> u:=vector([a,b,c,d]);
Now C is our matrix of interest.
> C:=evalm(u &* transpose(u));
We note that C has rank 1 just like A and hence will not be invertible
leading us to:
> det(C);
Page 4
>
rank(C);
>
nullspace(C);
We note that the output above makes sense as long as a0. If a=0 then
the first column of C will be all zeros so that [1,0,0,0] will be in N(C).
More generally, by multiplying each vector above by the scalar "a", the
nullspace will be spanned by [-d,0,0,a], [-c,0,a,0], and [-b,a,0,0] .
> N:= t -> evalm(t*id(4)-C);
>
det(N(t));
>
collect(%,t);
Note that there are really only two terms here, one which involves the
fourth power of t and the other which involves the third power of t. We
note that the matrix A had
. In fact this was the reason for
the scalar 64/85 which appears in front of the matrix defining A.
> solve(%=0,t);
>
N(%[4]);
>
nullspace(%);
Note that this just says that the eigenspace corresponding to the
nonzero eigenvalue is just Span{[a,b,c,d]}.
The following sequence of commands tries to determine what it will take
for the square of the matrix to be itself. The colon at the end of the
command suppresses output. I have only omitted it because it is lengthy
and not valuable to examine.
> F:=evalm(C^2-C):
> s:=seq(seq(F[i,j]=0,i=1..4),j=1..4):
> solve({s},{a,b,c,d});
Page 5
The output for this command is not given here because it is so lengthy.
However, in looking at all the solutions, each one indicates that for C^2-C
to be the zero matrix we must have the sums of the squares of a,b,c,d to
be 1. This illustrates one reason why 64/85 was used in defining the
matrix A.
Just to check this result, we try:
> G:=map(factor,F);
>
H:=matrix(4,4);
>
for j from 1 to 4 do for i from 1 to 4 do

H[i,j]:=subs(a^2=1-(b^2+c^2+d^2),G[i,j]) od od;
> evalm(H);
We can see from G that if

then G will be 0. This is
substantiated when we substitute
into each entry of G.
We see that C*C=C if and only if either all of a,b,c,d = 0 or the sum of the
squares of a,b,c,d is 1.
> trace(C);
>
assume(a,real);assume(b,real);assume(c,real);assume(d,real)
;
> norm(C,2);
Page 6
From this we see that the spectral norm of the matrix will be 1 if and only
if the sum of the squares of a,b,c,d is 1.
Also, in general, we see that if
where
the orthogonal projection onto the Span(u).
Page 7
, then A is the rank 1 matrix that is
Transformation Movie
Consider the linear transformation from
to
given by x
Ax where
We can examine this by looking at inputs and outputs, the images of those inputs under the linear
transformation. In the following movie the inputs are in red and the outputs are in blue. Each of the
T
inputs is a unit vector. The initial input is the vector (1 0) . It doesn't show because it is hiding on the
horizontal axis.
To restart the animation, just double click on it.

To stop the animation at any point, just click on it.
To restart it, double click again.
http://www.wooster.edu/math/linalg/movie.html2005/03/08 04:05:40 .
Basis Change
Effect of Change of Basis

Consider the linear transformation L:P
2x2
given by:
Matrix Representation With Respect To The Bases
From the last line above we see that the matrix A that represents L with respect to the bases E and F is given by:
http://www.wooster.edu/math/linalg/basischange.html (1 of 5)2005/03/08 04:06:02 .
Basis Change
Matrix Representation With Respect To the Bases
One should note that the lower right hand corner of the diagram above comes from the fact that:
From the last line in the diagram above we see that the matrix B that represents L with respect to the bases E and
F is given by:
Basis Change
Relationship Between A and B and Summary Diagram
In the picture above the transformations S ,T ,S ,T
E F E F
are the coordinate maps. The transformations S and T are
given by the transition matrices from the basis E to the basis E and the basis F to the basis F respectively. The
transistion matrices from the basis E to the basis E and from the basis F to the basis F are easy to constuct
because the bases E and F are standard bases. These two will give us S
Using the diagram we know that
B=TAS
Using the bases:
-1
-1
-1
and T .
Basis Change
we get that
-1
Putting this altogether using B=TAS
we get:
Basis Change
Definitions
LINEAR ALGEBRA DEFINITIONS

(Alphabetical Listing)
(List by Chapter)
Vectors
Systems of Linear Equations
Matrices
Linear Transformations
Vectors
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
n
Vector Addition in n
0 Vector in n
Scalar
Scalar Multiplication
Vector Subtraction
Dot Product in n
Vector Norm
Unit Vector
Standard Unit Vectors in n
Orthogonal Vectors
Orthonormal Vectors
Distance between Vectors
Angle between 2 Vectors
Linear Combination
Orthogonal Projection
Span
Spanning Set
Linear Independence
http://www.wooster.edu/math/linalg/defs.html (1 of 5)2005/03/08 04:06:21 .
Definitions
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
Linear Dependence
Subspace
Basis
Orthogonal Basis
Orthonormal Basis
Orthogonal Complement
Orthogonal Decomposition
Dimension
Coordinate Vector
Vector Space
Vector
Additive Identity
Additive Inverse
Infinite Dimensional Vector Space
C[a,b]
Pn
36. Mmn
37.
38.
39.
40.
41.
42.
43.
44.
F( )
Inner Product
Inner Product Space
Norm
Normed Vector Space
Uniform Norm
Taxicab Norm
Least Squares Approximation
Math 211 Home Page
Systems of Linear Equations

1. Linear Equation
2. System of Linear Equations
3. Solution
Definitions
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
Solution Set
Equivalent Systems
Homogeneous System
Consistent System
Underdetermined System
Overdetermined System
Augmented Matrix for a System
Coefficient Matrix
Free Variables
Lead Variables
Math 211 Home Page
Matrices
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Matrix
Matrix-Vector Multipliation
Lower Triangular
Upper Triangular
Triangular
Row Equivalent
Reduced Row Echelon Form
Elementary Matrix
Rank
Matrix Addition
Scalar Multiplication for a Matrix
Matrix Subtraction
Matrix Multiplication
Matrix Powers
Identity Matrix In
16. Nilpotent Matrix

17. Transpose
18. Symmetric
Definitions
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
Skew-symmetric
Column Space
Row Space
Null Space
Nullity
Invertible (or Nonsingular)
Singular (or Noninvertible)
Inverse
Pseudoinverse
Left Inverse
Right Inverse
Trace
Eigenvalue
Eigenvector
Eigenspace
Diagonally Dominant
Minor Submatrix
Cofactor
Determinant
Characteristic Polynomial
Algebraic Multiplicity
Geometric Multiplicity
Adjoint (Classical)
Similar
Orthogonal Matrix
Diagonalizable
Orthogonally Diagonalizable
QR Factorization
Dominant Eigenvalue
Matrix Norm
Frobenius Norm
Ill-conditioned
Singular Values
Singular Value Decomposition
Definitions
1.
2.
3.
4.
5.
6.
7.
8.
Linear Transformation
Kernel
Range
Matrix Representation
Matrix Transformation
Inverse Transformation
Isomorphism
Operator Norm
Math 211 Home Page
Math 211 Home Page
LINEAR ALGEBRA FACTSHEET

Invertibility of a Matrix
1)
A square matrix A is invertible iff null(A)={0}.
2)
A square matrix A is invertible iff rref(A)=I.
3)
An nxn square matrix A is invertible iff rank(A)=n.
4)
An square matrix A is invertible iff det(A) 0.
(1
"a b%
1 " d (b%
5)
$
' =
$
'
ad ( bc #(c a &
#c d&
6)
A matrix M has a right inverse iff MT has a left inverse.
!
7)
8)
9)
10)
A nonsquare matrix A has a left or right inverse if it is of full rank.

(i.e. if rank(A)=minimum(#of rows of A, #of columns of A))
A matrix has a left inverse iff its columns are linearly independent
An mxn matrix has a right inverse iff its rows span m.
An mxn matrix A has a right inverse iff rank(A) = m and has a left inverse iff rank(A)=n.
Subspaces
1)
The nullspace of an mxn matrix A, N(A) = {x n: Ax = 0}, is a subspace of n.
2)
R(A) = col(A) = {Ax: xn} for any mxn matrix A.
3)
The range of an mxn matrix A, R(A), is a subspace of m.
4)
xR(A)=col(A) iff x T " row( AT )
5)
xR(AT) =colspace(AT) iff xTrowspace(A)
N(A)=N(ATA)
!
Orthogonality
1)
If S is a subspace of n then S is a subspace of n.
2)
SS={0}.
3)
n=SS for any subspace S of n
dim(" n ) = n , dim( Pn ) = n + 1, dim( M mxn ) = mn , C[a,b] is infinite dimensional
4)
5)
If W=UV then UW={0}.
6)
If A is an mxn matrix then " n = null( A) # col( AT ) and " m = null( AT ) # col( A)
6)
! 7)
8)
9)
10)
11)
12)
13)
14)
15)
16)
! a subspace S of m then null( AT ) = S " = col( A) " .

If the!columns of A span
If A is an mxn matrix and xn and ym then <Ax,y>=<x,ATy> and x y = x T y .
The nonzero rows!of the rref(A) are a basis for!rowspace(A).
The transposes of the nonzero rows in rref(A)
are a basis for R(AT) .
!
Columns of A for which there are leading ones in rref(A) are linearly
independent and form a basis
!
for col(A).
dim(row(A))=dim(col(A))
P is the orthogonal projection onto col(P) iff P2=P and P=PT.
If s is the orthogonal projection of x onto a subspace S of n then s is the closest vector in S to x.
"1
If the columns of A are a basis for the subspace col(A) of n then P = A( AT A) AT is the orthogonal
projection onto S.
xy
The orthogonal projection of a vector x onto a vector y in n is given by p =
y.
y y
!
Linear Independence and Dimension

1)
Two vectors in a vector space are linearly independent iff neither is a scalar multiple of the other.
2)
A set of vectors in a vector space are linearly dependent iff one of them can be written as a linearly
combination of the others.
3)
If dim(V)=n and m>n then any collection of m vectors in V must be linearly dependent.
4)
If dim(V)=n and m<n then any collection of m vectors in V cannot span V.
5)
If dim(V)=n then any n independent vectors in V will also span V.
6)
If dim(V)=n then any n vectors that span V will also be linearly independent.
7)
If dim(V)=n and m<n then any set of m independent vectors can be extended to a basis for V. This is
done by first selecting a vector in V which is not in the span of the m independent vectors. Adding
this vector to the set of m vectors gives a set of m+1 vectors which will still be linearly independent.
Now repeat the process until n vectors are obtained.
8)
If A is a matrix then rank(A)+nullity(A)=# of columns of A.
9)
If A is a matrix then rank(A)=the number of leading ones in the rref(A) and nullity(A)=the number of
free variables in rref(A).
10)
Ax gives a linear combination of the columns of A for any xn.
11)
null(A)={0} iff the columns of A are linearly independent.
Inner Products and Norms
1)
For x,y in an inner product space we have <x,y>=||x||||y||cos() where is the angle between x and y.
2)
|<x,y>| ||x||||y|| for any x,y in an inner product space V. (Cauchy-Schwarz Inequality)
3)
For any x,y in an inner product space we have ||x+y|| ||x|| + ||y||. (Triangle Inequality)
4)
For any x in n and in we have ||x|| = || ||x||.
5)
To say M is a transition matrix from ordered basis E1 to ordered basis E2 means [v]E2 = M[v]E1 for
any vector v in the vector space.
6)
If [x1,x2, . . . ,xn] is an ordered basis for n then the transition matrix from this ordered basis to the
standard ordered basis is S = [ x 1 x 2 L x n ] .
7)
If [x1,x2, . . . ,xn] and [y1,y2, . . . ,yn] are two ordered bases for n then
the transition matrix from the first one to the second one is given by T-1S
where T = [ y 1 !y 2 L y n ] and S = [ x 1 x 2 L x n ] .
8)
If L:nm is a linear transformation then there is an mxn matrix A such that L(x)=Ax.
9)
If L:VW is a linear transformation with dim(V)=n and dim(W)=m and if E and F are ordered bases
for V and W respectively then there is an mxn matrix A such that [L(v)]F = A[v]E for all vV.
!
!
10)
xS, a subspace of n, iff x is orthogonal to each vector in any spanning set for S.
11)
tr(AB)=tr(BA) for any nxn matrices A and B.
12)
If A and B are similar matrices then tr(A)=tr(B).
Linear Algebra and Applications Textbook
Linear Algebra and Applications

Textbook
Home
Schedule
Teaching
Research
Service
Public Files
Welcome again. In order to enable prospective users to preview my text easily

and conveniently, I'm putting a copy of it on the web for your perusal. The
table of contents with links is at the bottom of this page. A few comments:
Personal
CV
Linear Algebra
Contact Me
Why this text? I'm committed to a balanced blend of theory, application and
computation. Mathematicians are beginning to see their discipline as more of
an experimental science, with computer software as the "laboratory" for
mathematical experimentation. I believe that the teaching of linear algebra
should incorporate this new perspective. My own experience ranges from pure
mathematician (my first research was in group and ring theory) to numerical
analyst (my current speciality). I've seen linear algebra from many viewpoints
and I think they all have something to offer. My computational experience
makes me like the use of technology in the course -- a natural fit for linear
algebra -- and computer exercises and group projects also fit very well into the
context of linear algebra. My applied math background colors my choice and
emphasis of applications and topics. At the same time, I have a traditionalist
streak that expects a text to be rigorous, correct and complete. After all, linear
algebra also serves as a bridge course between lower and higher level
mathematics.
Many thanks to those who have helped me in this project. In particular, John
Bakula for prodding me into moving this project into the final stages and
pointed me in the direction of McGraw-Hill custom publishing. They are
printing a nice soft copy text for a reasonable price to students -- about $26.
Thanks also to my colleagues Jamie Radcliffe, Lynn Erbe, Brian Harbourne,
Kristie Pfabe, Barton Willis and a number of others who have made many
suggestions and corrections. Thanks to Jackie Kohles for her excellent work
on solutions to the exercises.
http://www.math.unl.edu/~tshores/linalgtext.html (1 of 5)2005/03/08 04:09:38 .
About the process: I am writing the text in Latex. The pages you will see have
been converted to gif files for universal viewing with most browsers. The
downside of this conversion is that the pages appear at a fairly crude
resolution. I hope that they are still readable to all. Hardcopy of the text is
much prettier. Book form of the text can be purchased from McGraw-Hill
Primus Custom Publishing. The ISBN for the text is 0072437693. An errata
sheet for the text is provided below. In the near future, I will post a corrected
version of the text on this website, though I don't know when or if there will be
a published corrected version. If you have any suggestions or comments, drop
me a line. I appreciate any feedback.
Applied Linear Algebra and Matrix

Analysis
by
Thomas S. Shores
Copyright November 2003 All
Rights Reserved
Title Page
Preface
Chapter 1. LINEAR SYSTEMS OF EQUATIONS
1. Some Examples
2. Notations and a Review of Numbers
3. Gaussian Elimination: Basic Ideas
4. Gaussian Elimination: General Procedure
5. *Computational Notes and Projects
Review Exercises
Chapter 2. MATRIX ALGEBRA
1. Matrix Addition and Scalar Multiplication

2. Matrix Multiplication
3. Applications of Matrix Arithmetic
4. Special Matrices and Transposes
5. Matrix Inverses
6. Basic Properties of Determinants
7. *Applications and Proofs for Determinants
8. *Tensor Products
Review Exercises
Chapter 3. VECTOR SPACES
1. Definitions and Basic Concepts

2. Subspaces
3. Linear Combinations
4. Subspaces Associated with Matrices and Operators
5. Bases and Dimension
6. Linear Systems Revisited
7. *Change of Basis and Linear Operators
Review Exercises
Chapter 4. GEOMETRICAL ASPECTS OF STANDARD

SPACES
1. Standard Norm and Inner Product

2. Applications of Norm and Inner Product
3. Unitary and Orthogonal Matrices
5. Review Exercises
Chapter 5. THE EIGENVALUE PROBLEM
1. Definitions and Basic Properties

2. Similarity and Diagonalization
3. Applications to Discrete Dynamical Systems
4. Orthogonal Diagonalization
5. *Schur Form and Applications
6. *The Singular Value Decomposition
Review Exercises
Chapter 6. GEOMETRICAL ASPECTS OF ABSTRACT

SPACES
1. Normed Linear Spaces

2. Inner Product Spaces

3. Gram-Schmidt Algorithm
4. Linear Systems Revisited
5. *Operator Norms
Review Exercises
Appendix A.
Table of Symbols
Solutions to Selected Exercises
Bibliography
Index
Errata Sheet for the third edition in pdf format
[TOP][HOME][SCHEDULE][TEACHING]RESEARCH]
[UNL][A&S COLLEGE][MATH/STAT DEPT]
ATLAST Project
Welcome to the ATLAST Project Forum

The Second Edition of the ATLAST book is now available.
The second edition of ATLAST Computer Exercises for Linear Algebra is now available.
Instructors should contact their Prentice-Hall sales representatives to obtain copies and
ordering information. The ATLAST book can be used in conjunction with any Linear
Algebra textbook. The new edition contains more exercises and projects plus new and
updated M-files.
Special Deal - The ATLAST book is being offered as a bundle with the Linear Algebra
with Applications, 6th ed., by Steven J. Leon. The two book bundle is offered at the same
price as the Leon textbook alone, so the ATLAST book is essentially free! The ISBN for
the two book bundle is 0-13-104421-4.
Download the M-files for the second edition.
is a trademark for the MATLAB software distributed by the

Mathworks of Natick, MA.
The ATLAST M-files are add-on MATLAB programs. The M-files that accompany the
second edition of the ATLAST book are fully compatible with version 6.5 of MATLAB.
About a dozen new M-files were developed for the second edition. Click on the link
below to download a zip file containing the complete collection of M-files for the second
edition.
Download M-files for second edition.
http://www.umassd.edu/SpecialPrograms/Atlast/welcome.html (1 of 4)2005/03/08 04:10:12 .
ATLAST Project
About ATLAST
ATLAST is a National Science Foundation sponsored project to encourage and facilitate the use of
software in teaching linear algebra. The project has received the support of two NSF DUE grants as part
of their Undergraduate Faculty Enhancement program. The materials on this web page represent the
opinions of its authors and not necessarily those of the NSF.
The ATLAST project conducted eighteen faculty workshops during the six summers from 1992 to 1997.
The workshops were held at thirteen regional sites. A total of 425 faculty from a wide variety of colleges
and universities participated in the workshops.
Workshop participants were trained in the use of the MATLAB software package and how to use
software as part of classroom lectures. Participants worked in groups to design computer exercises and
projects suitable for use in undergraduate linear algebra courses. These exercises were class tested
during the school year following the workshop and then submitted for inclusion in a database. A
comprehensive set of exercises from this database covering all aspects of the first course in linear
algebra has been selected for a book ATLAST Computer Exercises for Linear Algebra. The editors of the
book are Steven Leon, Eugene Herman, and Richard Faulkenberry. The later ATLAST workshops
developed a series of lesson plans using software to enhance linear algebra classroom presentations.
These lesson plans were adapted into the exercise/project format used in the ATLAST book and
included in the second edition of the ATLAST book. The second edition of the ATLAST book is
available from Prentice Hall.
The ATLAST Project is coordinated through the University of Massachusetts Dartmouth. The ATLAST
Project Director is Steven Leon and the Assistant Director is Richard Faulkenberry. ATLAST
Workshops have been presented by Jane Day, San Jose State University, Eugene Herman, Grinnell
College, Dave Hill, Temple University, Kermit Sigmon, University of Florida, Lila Roberts, Georgia
Southern University, and Steven Leon.
Software
MATLAB
The software used for the ATLAST workshops has been MATLAB. The ATLAST organizers
believe that MATLAB is the software of choice for teaching linear algebra. A student version of
MATLAB is available from the MathWorks.
ATLAST Project
Mathematica
Mathematica versions of most of the exercises and projects in the first edition of the
ATLAST book are available. A collection of ATLAST Mathematica Notebooks has
been developed by Richard Neidingerof Davidson College. The collection of
notebooks can be downloaded from this link Mathematica Notebooks.
Maple
We are looking for somebody to develop Maple versions of the of the exercises and
projects in the second edition of the ATLAST book. If you are interested in doing this,
please contact Steve Leon at the address given at the bottom of this Web page.
Lesson Plans
A selection of the ATLAST lesson are available for download.
ATLAST Lesson Plans
The Past and the Future . . .
The History of the ATLAST project

List of the ATLAST contributors
Acknowledgements
Future Plans
If you have any questions, suggestions, or concerns about the ATLAST Project please contact:
Steven J. Leon
ATLAST Project Director
Department of Mathematics
University of Massachusetts Dartmouth
North Dartmouth, MA 02747-2300
ATLAST Project
Telephone: (508) 999-8320

Fax: (508) 910-6917
Email: [email protected]
or [email protected]
Originally created by (asb)
Linear Algebra Toolkit
2000
2002
P. Bogacki
Linear Algebra Toolkit Main Page
v. 1.23
This Linear Algebra Toolkit is comprised of the modules listed below. Each module is designed to help
a linear algebra student learn and practice a basic linear algebra procedure, such as Gauss-Jordan
reduction, calculating the determinant, or checking for linear independence.
Click here for additional information on the toolkit.
MODULES
Systems of linear equations and matrices
Row operation calculator
Interactively perform a sequence of elementary

row operations on the given m x n matrix A.
Transforming a matrix to row echelon form
Find a matrix in row echelon form that is row

equivalent to the given m x n matrix A.
Transforming a matrix to reduced row echelon

form
Find the matrix in reduced row echelon form

that is row equivalent to the given m x n matrix
A.
Solving a linear system of equations
Solve the given linear system of m equations in

n unknowns.
Calculating the inverse using row operations
Find (if possible) the inverse of the given n x n

matrix A.
Determinants
Calculating the determinant using row
operations
Calculate the determinant of the given n x n

matrix A.
Vector spaces
Linear independence and dependence
http://www.math.odu.edu/~bogacki/cgi-bin/lat.cgi (1 of 3)2005/03/08 04:10:34 .
Given the set S = {v1, v2, ... , vn} of vectors in

the vector space V, determine whether S is
linearly independent or linearly dependent.
Determining if the set spans the space

the vector space V, determine whether S spans
V.
Finding a basis of the space spanned by the set

the vector space V, find a basis for span S.
Finding a basis of the null space of a matrix
Find a basis of the null space of the given m x

n matrix A. (Also discussed: rank and nullity of
A.)
Linear transformations
Finding the kernel of the linear transformation
Find the kernel of the linear transformation

L: VW. (Also discussed: nullity of L; is L
one-to-one?)
Finding the range of the linear transformation
Find the range of the linear transformation

L: VW. (Also discussed: rank of L; is L onto
W?)
ADDITIONAL INFO
The Linear Algebra Toolkit has been written entirely in PERL. Every effort has been made to
make it compatible with a broad range of browsers, however, no guarantee can be made that
every browser will properly handle this application.
In order to bookmark the toolkit, first go to the cover page; all the other pages in this toolkit
(including the one you are reading) are created dynamically.
The toolkit represents all the data (such as matrix entries, vector components and polynomial
coefficients) as rational numbers, where both the numerator and denominator are stored as
integers. All operations on these rational numbers are performed exactly. Note that in the current
version overflow errors are not trapped.
Permission is hereby granted for any noncommercial use of this product by individuals, e.g.
students, faculty, etc.
Permission for the class use of this product in an academic environment is automatically granted,
provided the instructor notifies the author.
Any other use (including any commercial use) requires an explicit permission from the author.
Comments should be forwarded to the author:

Przemyslaw Bogacki.
Department of Mathematics and Statistics
Old Dominion University
Norfolk, VA 23529
Phone: (757) 683-3262
E-mail: [email protected]
You can read testimonials from students and instructors who used the Toolkit.
These pages should be considered work in progress. The interface and functionality will
undergo changes as new versions are released.
Here is a list of all released versions:
Version
Date released
Description
1.00
May 6, 2000
Row Operation Calculator
1.20
ROC becomes Linear Algebra

September 6, 2000 Toolkit
5 modules added
1.21
October 17, 2000
2 modules added
1.22
October 26, 2000
First official (non-beta) release
1.22a
November 27, 2000 Bug fixes
1.23
October 25, 2002
2 modules added
Elements of Abstract and Linear Algebra

Edwin H. Connell
This is a foundational textbook on abstract algebra with emphasis on linear
algebra. You may download parts of the book or the entire textbook. It is
provided free online in PDF, DVI, postscript, and gzipped postscript.
Please read some words from the author first.
Join the revolution in education! Write a supplement to this book and put it online.
Participate in and contribute to a forum on abstract and linear algebra. Ask questions, give
answers, and make comments in a worldwide classroom with the internet as blackboard.
[March 20, 2004]
The entire book

Title page and copyright
Introduction
Outline
Chapter 1: Background and Fundamentals of Mathematics
Chapter 2: Groups
Chapter 3: Rings
Chapter 4: Matrices and Matrix Rings
Chapter 5: Linear Algebra
Chapter 6: Appendix
http://www.math.miami.edu/~ec/book/ (1 of 2)2005/03/08 04:11:04 .
Index
Please send comments to Edwin H. Connell

Web page created by Dmitry Gokhman
http://www.math.miami.edu/~ec/book/ (2 of 2)2005/03/08 04:11:04 .
Linear Algebra textbook home page
Linear Algebra
by
Jim Hefferon
Mathematics
Saint Michael's College
Colchester, Vermont USA 05439
My text Linear Algebra is free for downloading, It covers the material of
an undergraduate first linear algebra course. You can use it either as a main
text, or as a supplement to another text, or for independent study.
What's Linear Algebra About?

When I started teaching linear algebra I found three kinds of texts. There were applied mathematics
books that avoid proofs and cover the linear algebra only as needed for their applications. There were
advanced books that assume that students can understand their elegant proofs and understand how to
answer the homework questions having seen only one or two examples. And there were books that
spend a good part of the semester doing elementary things such as multiplying matrices and computing
determinants and then suddenly change level to working with definitions and proofs.
Each of these three types was a problem in my classroom. The applications were interesting, but I
wanted to focus on the linear algebra. The advanced books were beautiful, but my students were not
ready for them. And the level-switching books resulted in a lot of grief. My students immediately
thought that these were like the calculus books that they had seen before, where there is material labelled
`proof' that they have successfully skipped in favor of the computations. Then, by the time that the level
switched, no amount of prompting on my part could convince them otherwise and the semesters ended
unhappily.
That is, while I wish I could say that my students now perform at the level of the advanced books, I
cannot. However, we can instead work steadily to bring them up to it, over the course of our program.
This means stepping back from rote computations of the application books in favor of an understanding
of the mathematics. It means proving things and having them understand, e.g., that matrix multiplication
is the application of a linear function. But it means also avoiding an approach that is too advanced for
the students; the presentation must emphasize motivation, must have many exercises, and must include
problem sets with many of the medium-difficult questions that are a challenge to a learner without being
overwhelming. And, it means communicating to our students that this is what we are doing, right from
http://joshua.smcvt.edu/linearalgebra/ (1 of 4)2005/03/08 04:11:18 .
the start.
Summary Points
The coverage is standard: linear systems and Gauss' method, vector spaces, linear maps and
matrices, determinants, and eigenvectors and eigenvalues. The Table of Contents gives you a
quick overview.
Prerequisites A semester of calculus. Students with three semesters of calculus can skip a few
sections.
Applications Each chapter has three or four discussions of additional topics and applications.
These are suitable for independent study or for small group work.
What makes this book different? Its approach is developmental. Although the presentation is
focused on proving things and covering linear algebra, it does not start with an assumption that
students are already able at abstract work. Instead, it proceeds with a great deal of motivation,
and many examples and exercises that range from routine verifications to (a few) challenges. The
goal is, in the context of developing the usual material of an undergraduate linear algebra course,
to help raise the level of mathematical maturity of the class.
There is only one review that I know of.
Here Is Linear Algebra

The files are current as of 2003-May-05. To have a look at the PDF files you need Adobe Acrobat. (Note
for those with limited Internet access: this material is available as part of the eGranary Project; perhaps
that will help you)
The whole book Linear Algebra along with the answers to all exercises or just the answers to
checked exercises. If you save the two files in the same directory then clicking on an exercise
will send you to its answer and clicking on an answer will send you to its exercise. (You will
need to rename the file "jhanswer.pdf" to use the hyperlink feature. Note Natalie Kehr has
sent me a suggestion for people wanting to access the answers in this way. "In Acrobat reader go
to Edit | Preferences | General and uncheck the box for Open cross-document links in same
window. Also, a possible reasonable way of working is to click on the hyperlink to get from the
question to the answer, but to return by closing the answer window. By using hyperlinks both
ways, without realizing it I eventually landed up with dozens of open windows. Another good
technique is to use a single hyperlink and then use Alt W 1 and Alt W 2 to toggle between the
two windows.")
The first chapter, on linear elimination.
The second chapter, on vector spaces.
The third chapter, on linear maps and matrix operations.

The fourth chapter on determinants.
The fifth chapter, on matrix similarity and Jordan form.
The source. You need to know LaTeX and MetaPost to work with it, although there is a readme
file to get you going. Also there is a bit of optional material, and a version of the book and
answers in PostScript that has been zipped.
If you just want a quick look, try the second chapter. The first chapter may be misleading because it is
necessarily computational but the second chapter shows clearly what the book works on: bridging
between high school mathematics with its reliance on explicitly-given algorithms and upper division
college mathematics with its emphasis on concepts and proof.
Naturally, I'd be glad for any comments or interest in trying it out. It's been class-tested here and
elsewhere, and I'm delighted to have it on student's desks, doing what it is for. Around here, running off
double-sided copies and comb-binding them costs students $20.00 (we include the Preface, Table of
Contents, Appendix, and Index to make 449 pages. With the Answers to Exercises at 213 pages the total
is $25.00).
Can You Help With Linear Algebra?

In addition to the book, the answers to the exercises, and the source, you can get some materials that
were contributed by others. If you in turn have something that you are able to share please write me.
Anything is welcome, from a suggestion about a better way to prove a result or make an example, to an
entire Topic or subsection. In particular, your used exams or problem sets would be very welcome,
especially if you can contribute the TeX or LaTeX source, so that a user could cut or paste. Several
people have asked me about a Topic on eigenvectors and eigenvalues in Quantum Mechanics. Sadly, I
don't know any QM. Can you help?
One point about contacting me with typo and bug reports: I receive such reports from many people and
they are a big help. I save them and once in a while do a batch of updates, so if you made a report and it
hasn't gotten addressed yet, please be patient. Conversely, for those people who have gotten these
materials on a disk, please be aware that there is probably a more recent version on-line and you can
check there for updates, if possible.
Naturally, all contributions will be gratefully acknowledged (or anonymous, if you like).
Has Linear Algebra Helped You?

If you find the book useful, drop me a note in the mail. In particular, if you use it in a class, a letter on
letterhead to the address below would be great.
Jim Hefferon
Mathematics, Saint Michael's College
Winooski Park
Colchester, VT USA, 05439
Some people don't believe me when I tell them that I'm working in here.
Page Information
The page above. The page maintainer is Jim Hefferon.
This site Joshua is located in the Mathematics Department of Saint Michael's College in Colchester,
Vermont USA. Joshua runs under Linux. Open Source software is a great idea. This project would not
have gotten done without it.
(Credit for the logo to Matt Ericson.)
ELEMENTARY LINEAR ALGEBRA
Elementary Linear Algebra

(Lecture Notes by Keith Matthews, 1991)
CMAT: An exact arithmetic matrix calculator program
Preface
Title Page/Contents (pages 0, i-iv)
pdf version of book (731K)
Chapter 1: Linear Equations (pages 1-21)
Chapter 2: Matrices (pages 23-54)
Chapter 3: Subspaces (pages 55-70)
Chapter 4: Determinants (pages 71-88)
Chapter 5: Complex Numbers (pages 89-114)
Chapter 6: Eigenvalues and Eigenvectors (pages 115-128)
Chapter 7: Identifying Second Degree Equations (pages 129-148)
Chapter 8: Three-dimensional Geometry (pages 149-187)
Further Reading/Bibliography (pages 189,191-193)
Index (pages 194-196)
Corrections
Solutions to Elementary Linear Algebra

(Prepared by Keith Matthews, 1991)
Title Page/Contents (pages 0/i)

pdf version of the solutions (437K - best read with zoom in)
Problems 1.6: Linear Equations (pages 1-11)
Problems 2.4: Matrices (pages 12-17)
Problems 2.7: Matrices (pages 18-31)
Problems 3.6: Subspaces (pages 32-44)
Problems 4.1: Determinants (pages 45-57)
Problems 5.8: Complex Numbers (pages 58-68)
Problems 6.3: Eigenvalues and Eigenvectors (pages 69-82)
Problems 7.3: Identifying Second Degree Equations (pages 83-90)
http://www.numbertheory.org/book/ (1 of 2)2005/03/08 04:11:22 .
ELEMENTARY LINEAR ALGEBRA
Problems 8.8: Three-dimensional Geometry (pages 91-103)
Last modified 10th December 2004
http://www.numbertheory.org/book/ (2 of 2)2005/03/08 04:11:22 .
Lee Lady: Notes on Linear Algebra
Notes on Linear Algebra

Professor Lee Lady
University of Hawaii
Mostly there are two themes in this collection of short hand-outs. First, that rather than
think of an m by n matrix as a doubly-indexed array, it is often more enlightening to think
of it as a n-tuple of columns (which are, of course, m-vectors) or an m-tuple of rows.
And secondly, that once one understands the method of solving systems of equations by
elimination, essentially one knows the whole of the linear algebra covered in this (very
stripped-down) course. Everything else, except for eigenvectors and eigenvalues, is just a
matter of constantly restating the same facts in different language.
A lot of the files listed below are in PDF (Adobe Acrobat) format. Alternate versions are in DVI
format (produced by TeX; see see here for a DVI viewer provided by John P. Costella) and
postscript format (viewable with ghostscript.) Some systems may have some problem with certain
of the documents in dvi format, because they use a few German letters from a font that may not be
available on some systems. (Three alternate sites for DVI viewers, via FTP, are CTAN, Duke, and
Dante, in Germany.)
Systems of Linear Equations in a Nutshell

(Click here for dvi format.)
(Click here for Postscript format.)
Instead of thinking of a systems of equations as constituting m equations in n unknowns,

where all the coefficients are scalars, it can be more enlightening to think of it as a single
equation in n unknowns where the coefficients (and constant term) are m-dimensional
vectors.
http://www.math.hawaii.edu/%7Elee/linear/ (1 of 4)2005/03/08 04:11:39 .
How to Find the Inverse of a Matrix

(Click here for Postscript format format.)
Doing an elementary row operation on the left-hand factor A of a matrix product AB gives
the same result as doing the same operation on the product matrix. Using this observation,
it is easy to explain why the usual process for inverting a matrix works, and why the left
inverse and the right inverse are identical.
This approach enables one to omit the topic of elementary matrices from the course.
Some Equivalent Statements

Some Equivalent Characterizations of Basic Concepts

A ``Grammar Lesson'' in Linear Algebra

Some incorrect statements frequently found in student proofs.
The Pivotal Role of Zero in Linear Algebra

The Logical Structure of Proving Linear Independence

Students seem to have enormous difficulty in learning the pattern for proving any
statement that essentially reduces to an implication, such as proving that vectors are
linearly independent or that a function is one-to-one. (This may also be a main source of
the difficulty students have with proofs by induction.)
When asked to prove "If P, then Q," students will almost invariably begin by saying,
"Suppose Q."
The logical analysis here was one of my attempts to clarify this type of proof for students.
I don't know whether it actually helps or not.
The Column Space of a Matrix

By definition, the column space of an m by n matrix A with entries in a field F is the

subspace of Fm spanned by the columns of A. A close examination of the method of
elimination shows that a basis for this space can be obtained by choosing those columns of
A which will contain the leading entries of rows after A is reduced to row-echelon form.
(The row echelon form of A shows which columns to choose, but the basis columns
themselves must come from the original matrix A.)
Eigenvalues
Suppose than an n by n matrix A has n linearly independent eigenvectors and let P be the
matrix whose columns are these eigenvectors. Then the jth column of the product AP is
readily seen to be equal to the jth column of P multiplied by the jth eigenvalue. If now we
write Q for the inverse of P, it follows easily that QAP is a diagonal matrix with the
eigenvalues on the diagonal.
This approach is not dependent on change-of-basis formulas.
Syllabus for Spring, 1996
[ Top of Page | Calculus | HOME ]
This page has been accessed
times
ELECTRONIC Journal of
LINEAR ALGEBRA
Published by
ILAS - The International Linear Algebra Society
ISSN 1081-3810
Israel
Primary Site:
Mirror Sites:
USA
Germany
Portugal
EMIS
Editors in Chief:
Ludwig
Elsner
Daniel
Hershkowitz
Advisory Editors:
Ravindra B.
Bapat
Peter
Lancaster
Richard A.
Brualdi
Hans
Schneider
Miroslav
Fiedler
Bryan L.
Shader
Shmuel
Friedland
Daniel B.
Szyld
Stephen J.
Kirkland
Associate Editors:
Harm Bart
Robert Guralnick
Managing Editor:
http://hermite.cii.fc.ul.pt/iic/ela/ (1 of 4)2005/03/08 04:12:13 .
Michael Neumann Bit-Shun Tam
Other EMIS sites
Daniel Hershkowitz
Associate Managing Editor:

Michael J. Tsatsomeros
Assistant Managing Editor:

Shaun Fallat
Editors' e-mail addresses
Editors' Postal addresses
Former editors
Volume 14 (2005)
Table of Contents
Articles
Author Index
Volume 13 (2005)
Table of Contents
Articles
Author Index
Volume 12 (2004/2005)
Table of Contents
Articles
Author Index
Volume 11 (2004)
Table of Contents
Articles
Author Index
Volume 10 (2003)
Table of Contents
Articles
Author Index
Volume 9 (2002)
Table of Contents
Articles
Author Index
Volume 8 (2001)
Table of Contents
Articles
Author Index
Volume 7 (2000)
Table of Contents
Articles
Author Index
Volume 6 (1999/2000)
Table of Contents
Articles
Author Index
Volume 5 (1999)
Table of Contents
Articles
Author Index
Volume 4 (1998)
Table of Contents
Articles
Author Index
Volume 3 (1998)
Table of Contents
Articles
Author Index
Volume 2 (1997)
Table of Contents
Articles
Author Index
Volume 1 (1996)
Table of Contents
Articles
Author Index
Multi-year Author Index:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
ELA guide:
preparing an article
submitting an article
reading and downloading
ELA Information about:

the Electronic Journal
Copyrights Transfer Agreement
PRINT VERSION OF ELA

ELA-LIST
IIC - ILAS Information Center
Click
to send your comments to ELA
Your visit here is number

Since March 1, 1999
Links2Go
Mathematical Journals
Down with Determinants!

Sheldon Axler
This paper was published in the American Mathematical Monthly 102 (1995), 139-154.
In 1996 this paper received the Lester R. Ford Award for expository writing from the Mathematical
Association of America.
Abstract: This paper shows how linear algebra can be done better without determinants. The standard
proof that a square matrix of complex numbers has an eigenvalue uses determinants. The simpler and
clearer proof presented here provides more insight and avoids determinants. Without using determinants,
this allows us to define the multiplicity of an eigenvalue and to prove that the number of eigenvalues,
counting multiplicities, equals the dimension of the underlying space. Without using determinants, we
can define the characteristic and minimal polynomials and then prove that they behave as expected. This
leads to an easy proof that every matrix is similar to a nice upper-triangular one. Turning to inner
product spaces, and still without mentioning determinants, this paper gives a simple proof of the finitedimensional spectral theorem.
To obtain the entire paper, select your choice below:
pdf version
dvi version
PostScript version
My book Linear Algebra Done Right is partly based on ideas from this paper. The book is intended to be
a text for a second course in linear algebra.
http://www.axler.net/DwD.html (1 of 2)2005/03/08 04:12:34 .
Sheldon Axler's publications
Sheldon Axler's home page
http://www.axler.net/DwD.html (2 of 2)2005/03/08 04:12:34 .
Linear Algebra Glossary

This file defines common terms from linear algebra.
TABLE OF CONTENTS
A-Orthogonal Vectors
Adjacency Matrix
Adjoint Matrix
Alternating Sign Matrix
Anticirculant Matrix
Antisymmetric Matrix
Band Matrix
Band Matrix Storage
Bandwidth
Basis
Bidiagonal Matrix
Basic Linear Algebra Subprograms (BLAS)
Block Matrix
Border Banded Matrix
Cartesian_Basis_Vectors
The Cauchy-Schwarz Inequality
The Cayley-Hamilton Theorem
CentroSymmetric Matrix
Characteristic Equation
Cholesky Factorization
Circulant Matrix
Cofactor Matrix
Column Echelon Form
Commuting Matrices
Companion Matrix
Compatible Norms
Complex Number Representation
Condition Number
Congruent Matrix
Conjugate Gradient Method
http://www.psc.edu/%7Eburkardt/papers/linear_glossary.html (1 of 151)2005/03/08 04:13:36 .
Conjugate Matrix
Conjugate of a Complex Number
Conjunctive Matrix
Convergent Matrix
Cross Product
Cyclic Reduction
Cyclic Tridiagonal Matrix
Defective Matrix
Deflation
Derogatory Matrix
Determinant of a Matrix
Diagonal Dominance
Diagonal Matrix
Diagonalizable Matrix
Downshift Matrix
Eigenvalues
Eigenvectors
EISPACK
EISPACK Matrix Norm
Elementary Matrix
Ellipsoids
Equilibration
Equivalent Matrix
Exchange Matrix
External Storage Algorithms
Fourier Matrix
Frobenius Matrix Norm
Gauss Elimination
Gauss Jordan Elimination
Gauss Seidel Iteration For Linear Equations
General Matrix Storage
Generalized Permutation Matrix
Gershgorin Disks
Givens Rotation Matrix
Gram Schmidt Orthogonalization
Hadamard's Inequality
Hadamard Product
Hankel Matrix
Harwell Boeing Sparse Matrix Collection (HBSMC)
HBSMC Sparse Matrix Storage
HBSMC Finite Element Matrix Storage
Hermite Normal Form
Hermitian Matrix
Hessenberg Matrix
Householder Matrix
Idempotent Matrix
The Identity Matrix
Ill Conditioned Linear System
The Incomplete LU Factorization
Inertia
Inner Product
Inverse Matrix
Inverse Power Method
Invertible Matrix
Involutory Matrix
Irreducible Matrix
Iterative Methods for Eigenvalues
Iterative Methods for Linear Equations
Iterative Refinement
The Jacobi Algorithm for Eigenvalues
The Jacobi Algorithm for Linear Equations
The Jacobi Preconditioner
Jordan Canonical Form
L1 Matrix Norm
L1 Vector Norm
L2 Matrix Norm
L2 Vector Norm
L Infinity Matrix Norm
L Infinity Vector Norm
LAPACK
LDL Factorization
Linear Dependence
Linear Least Squares Problems
Linear Space
LINPACK
LU Factorization
M Matrix
Magic Square
Matrix Exponential
Matrix Factorization
Matrix Norm
Matrix_Order
Matrix Properties
Matrix Rank
Matrix Splitting
Matrix Square Root
Matrix Storage
Matrix Structure
Matrix Symmetry
Minimal Polynomial
Minor Matrix
Monic Polynomial
Multiplicity
Nilpotent Matrix
Nonnegative Matrix
Normal Equations
Normal Matrix
Null Space
Null Vector
Orthogonal Matrix
Orthogonal Similarity Transformation
Orthonormal Vectors
Outer Product
Orthogonal Projector Matrix
Permanent of a Matrix
Permutation Matrix
Perron-Frobenius Theorem
Persymmetric Matrix
Pivoting
Polar Decomposition
Positive definite matrix
Positive Matrix
The Power Method
Preconditioner
Property A
The Pseudoinverse
QR Factorization
QR Method for Eigenvalues
Quaternion Representation
Rayleigh Quotient
Rectangular Matrix
Reflection Matrix
Residual Error
Root of Unity
Rotation
Row Echelon Form
Row Rank
Row Space
The Schur Decomposition
Sherman Morrison Formula
Sign Symmetric Matrix
Similar Matrix
Singular Matrix
Skew CentroSymmetric Matrix
Skew Hermitian Matrix
Skyline Matrix Storage
Span of a Set of Vectors
The Sparse BLAS
Sparse Matrix
Sparse Matrix Storage
SPARSKIT
Spectral Radius
Spectrum
Square Matrix
Stochastic Matrix
Strassen's Algorithm
Submatrix
Successive Overrelaxation Method (SOR)
Symmetric Matrix
Symmetric Matrix Storage
TESTMATRIX
Toeplitz Matrix
Trace of a Matrix
Transpose
Trapezoidal Matrix
Triangular Matrix
Tridiagonal Matrix
Unimodular Matrix
Unitary Matrix
Unitary Similarity Transformation
Upshift Matrix
Vector-Bound Matrix Norm
Vector Norm
Zero Matrix
Zero One Matrix
A-Orthogonal Vectors
Two vectors u and v are said to be A-orthogonal if
( u, A * v ) = 0.
Here A should be a positive definite symmetric matrix, which in turn guarantees that the expression ( u,
A * v ) may be regarded as an inner product of the vectors u and v, with the usual properties.
This concept is useful in the analysis of the conjugate gradient method.
Back to TABLE OF CONTENTS.
Adjacency Matrix
An adjacency matrix of an (undirected) graph is a matrix whose order is the number of nodes, and
whose entries record which nodes are connected to each other by a link or edge of the graph.
If two nodes I and J are connected by an edge, then Ai,j=1. All other entries of the matrix are 0. Thus, an
adjacency matrix is a zero-one matrix. The usual convention is that a node is not connected to itself, and
hence the diagonal of the matrix is zero.
The product A2=A*A is a matrix which records the number of paths between nodes I and J. If it is
possible to reach one node from another, it must be possible in a path of no more than n-1 links. Hence,
the reachability matrix, which records whether it is possible to get from node I to node J in one or more
steps, can be determined by taking the logical sum of the matrices I, A, A2, ..., An-1.
Adjoint Matrix
The adjoint matrix of a square matrix A has the property that:
A * adjoint ( A ) = adjoint ( A ) * A = det(A) * I.
Thus, the adjoint of A is "almost" the inverse. If A is invertible, then the inverse of A, denoted inverse
( A ) can be written explicitly as:
A-1 = ( 1 / det(A) ) * adjoint ( A ).
The adjoint matrix, in turn, is defined in terms of the cofactor matrix of A:
adjoint ( A ) = ( cofactor ( A ) )T.
Alternating Sign Matrix

An alternating sign matrix is an integer matrix with the properties that
the entries are only 0, +1, or -1;

the sum of the entries in each row and column is 1;
the first (and last) nonzero entry in each row and column is 1;
the nonzero entries in any row or column alternate in sign.
Example:
0
1
0
0
0
1
-1
1
0
0
0
0
0
0
1
0
1
-1
1
0
0
0
1
0
0
Obviously, alternating sign matrices include the identity matrix and any permutation matrix. From the
definitions, you should see that the first row must contain a single entry which is 1, with the other values
being 0. The nonzeroes in the second row can be a single 1, or the values, 1, -1, 1, in that order, with
some intervening zeroes possible. In the third row, the value -1 may occur up to as many times as there
were 1's in preceding rows, which means the most interesting row could be 1, -1, 1, -1, 1, -1, 1. Thus the
number of possible nonzeroes grows until the central row of the matrix is reached. Since the same
restrictions apply from the bottom reading up, the number of possible nonzeroes must now decrease.
Similar reasoning controls the nonzero population of the columns.
If we let An denote the number of distinct alternating sign matrices of order n, then it has only recently
been proved that
An = Product ( 0 <= I <= N-1 ) (3*I+1)! / (N+I)!
giving the sequence 1, 2, 7, 42, 429, 7436, 218348, 10850216, ...
Reference:
David Robbins,
The Story of 1, 2, 7, 42, 429, 7436, ...,
Mathematical Intelligencer,
Volume 13, Number 2, pages 12-19.
David Bressoud,
Proofs and Confirmations: The Story of the Alternating Sign Matrix Conjecture
Cambridge University Press, 1999.
Anticirculant Matrix
An anticirculant matrix is a matrix whose first row of values is repeated in each successive row, shifted
one position to the left, with the first value "wrapping around" to the end.
Here is an example of an anticirculant matrix:

1
2
3
4
2
3
4
1
3
4
1
2
4
1
2
3
and here is an example of a rectangular anticirculant matrix:

1 2 3 4 5
2 3 4 5 1
3 4 5 1 2
Simple facts about a anticirculant matrix A:
A is constant along any antidiagonal;

A is a special kind of Hankel matrix;
If A is square, then A is normal, hence unitarily diagonalizable.
If A is square, then the vector (1,1,...,1) is an eigenvector of the matrix, with eigenvalue equal to
the sum of the entries of a row.
Antisymmetric Matrix
A square matrix A is antisymmetric if it is equal to the negative of its transpose:
A = - AT.
Every matrix A can be decomposed into the sum of an antisymmetric and a symmetric matrix:
A = B + C = (1/2) * ( ( A - AT ) + ( A + AT ) )
Simple facts about an antisymmetric matrix A:
A has a zero diagonal;

A has pure imaginary eigenvalues;
A is normal, hence unitarily diagonalizable.
I - A is not singular;
The matrix ( I + A ) * Inverse ( I - A ) is orthogonal;
An antisymmetric matrix is also called skew symmetric.

In complex arithmetic, the corresponding object is a skew Hermitian matrix.
Band Matrix
A band matrix is a matrix whose entries are all zero except for the diagonal and a few of the
immediately adjacent diagonals.
If the band matrix is large enough, then many significant efficiencies can be achieved in storage and
matrix operations. Because so many elements are zero, the band matrix can be stored more compactly
using band matrix storage. And because so many elements are zero, many algorithms can be speeded up
to execute more quickly, including matrix multiplication and Gauss elimination.
Special cases include band matrices which are symmetric, positive definite, or tridiagonal.
Here is an example of a band matrix:
11
21
31
0
0
0
12
22
32
42
0
0
0
23
33
43
53
0
0 0 0
0 0 0
34 0 0
44 45 0
54 55 56
64 65 66
This matrix has an upper bandwidth of 1, and lower bandwidth of 2, and an overall bandwidth of 4.
LAPACK and LINPACK include special routines for a variety of band matrices. These routines can
compute the LU factorization, determinant, inverse, or solution of a linear system.
LAPACK and EISPACK have routines for computing the eigenvalues of a symmetric banded matrix.
Band Matrix Storage

Band matrix storage is a matrix storage format suitable for efficiently storing the nonzero entries of a
band matrix.
Most band storage schemes are column oriented. The nonzero entries of the matrix "slide" downwards,
while remaining in their original column. If the original matrix looked like this:
11
21
31
0
0
0
12
22
32
42
0
0
0
23
33
43
53
0
0 0 0
0 0 0
34 0 0
44 45 0
54 55 56
64 65 66
the matrix would be saved in column band matrix storage as:
0
11
21
31
12
22
32
42
23
33
43
53
34 0 0
44 45 0
54 55 56
64 65 66
Note that the zeroes in the above array are there just as padding. They don't correspond to any entries of
the original array, and are simply necessary to make the array rectangular.
If the matrix is to be handled by a Gauss elimination routine that uses pivoting, then there is a possibility
of fill in; that is, nonzero entries may need to be stored in places where zeroes had been. Band matrix
storage can still be used, but we need to include in the compressed matrix some extra entries
representing the diagonals along which the fill in entries may occur. It turns out that the number of extra
diagonals required is simply the number of nonzero subdiagonals in the original matrix. For our
example, this would mean the matrix would be stored as:
0
0
0
11
21
31
0
0
12
22
32
42
0
0
23
33
43
53
0 0 0
0 0 0
34 0 0
44 45 0
54 55 56
64 65 66
Bandwidth
The bandwidth of a band matrix is, roughly speaking, the number of diagonals that contain nonzero
entries.
More precisely, define ML, the lower bandwidth of a matrix A to be the maximum value of ( I - J ), and
MU to be the maximum value of ( I - J ), for all nonzero matrix entries A(I,J). Then the bandwidth M is
defined by:
M = ML + 1 + MU.
This definition always treats the (main) diagonal as nonzero, and is not misled by a matrix which has
only two nonzero diagonals, which are actually widely separated. All the territory between the diagonals
must be included when measuring bandwidth.
Basis
A basis for a linear space X of dimension N is a set of N vectors, {v(i) | 1 <= i <= N } from which all the
elements of X can be constructed by linear combinations.
Naturally, we require that each of the vectors v(i) be an element of the space X. Moreover, it is not
enough that these vectors span the space; we also require that they be linearly independent, that is, there
should be no redundant vectors in the set.
The columns of the identity matrix form a basis for the linear space of vectors of dimension N. A square
matrix of order N is not defective exactly when its eigenvectors form a basis for the linear space of
vectors of dimension N.
Given a particular basis, the representation of a vector x is the unique set of coefficients c(i) so that
x = Sum ( 1 <= I <= N ) c(i) * v(i)
The coefficients must be unique, otherwise you can prove that the basis is not linearly independent!
If the basis vectors are pairwise orthogonal, then the basis is called an orthogonal basis. If the basis
vectors have unit length in the Euclidean norm, the basis is a normal basis. If both properties apply, it is
an orthonormal basis. The columns of an orthogonal matrix are an orthonormal basis for the linear space
of vectors of dimension N.
Bidiagonal Matrix
A bidiagonal matrix has only two nonzero diagonals. The matrix is called upper bidiagonal if these are
the main diagonal and the immediate upper diagonal. The matrix is called lower bidiagonal if these are
the main diagonal and the immediate lower diagonal.
A simple example of an upper bidiagonal matrix is:
1
0
0
0
0
2
3
0
0
0
0
4
5
0
0
0
0
6
7
0
0
0
0
8
9
The Jordan Canonical Form is an example of an upper bidiagonal matrix.

A bidiagonal matrix is automatically a:
tridiagonal matrix;
triangular matrix;
band matrix.
with all the rights and privileges appertaining thereunto.

Basic Linear Algebra Subprograms (BLAS)

The BLAS or Basic Linear Algebra Subprograms, are a set of routines offering vector and matrix
utilities. They are extensively used as part of LINPACK and LAPACK, to simplify algorithms, and to
make them run more quickly.
The Level 1 BLAS provide basic vector operations such as the dot product, vector norm, and scaling.
Level 2 BLAS provide operations involving a matrix and a vector, and Level 3 BLAS provide matrixmatrix operations.
There are also sets of sparse BLAS and parallel BLAS available.
Here are the Level 1 BLAS routines for real, single precision vectors:
ISAMAX returns the index of maximum absolute value in vector SX;

SASUM computes the sum of absolute values of vector SX;
SAXPY adds a scalar multiple of one vector to another;
SCOPY copies vector SX into SY;
SDOT computes the dot product of two vectors;
SMACH estimates the largest and smallest machine values, and roundoff value;
SNRM2 computes the Euclidean norm of a vector SX;
SROT applies a Givens rotation;
SROTG generates a Givens Rotation;
SSCAL scales a vector by a constant;
SSWAP interchanges two vectors SX and SY.
Block Matrix
A block matrix is a matrix which is described as being built up of smaller matrices.
For example, a tridiagonal block matrix might look like this:
2 4 | 3 9 | 0 0
4 6 | 0 3 | 0 0
--------------1 0 | 2 4 | 3 9
5 5 | 4 6 | 0 3
--------------0 0 | 1 0 | 2 4
0 0 | 5 5 | 4 6
but for certain purposes, it might help us to see this matrix as "really" being a tridiagonal matrix, whose
elements are themselves little matrices:
a | b | 0
--------c | a | b
--------0 | c | a
An algorithm suitable for a tridiagonal matrix can often be extended, in a natural manner, to handle a
block tridiagonal matrix. Similar extensions can be made in some cases for other types of block
matrices. A block banded matrix can be factored by a variant of banded Gauss elimination, for instance.
Border Banded Matrix

A border banded matrix is a 2 by 2 block matrix comprising a (large) leading block which is a square
banded matrix, two dense rectangular side strips, and a (small) trailing block which is a square dense
matrix.
For example, a "toy" border banded matrix might look like this:
2 -1 0 0 0 0 0 | 1 2
-1 2 -1 0 0 0 0 | 2 5
0 -1 2 -1 0 0 0 | 7 8
0 0 -1 2 -1 0 0 | 3 3
0 0 0 -1 2 -1 0 | 4 2
0 0 0 0 -1 2 -1 | 3 1
0 0 0 0 0 -1 2 | 7 8
------------------------3 7 8 3 2 3 1 | 5 2
1 2 4 7 9 2 4 | 3 6
which we can regard as being made up of the blocks:

A11 | A12
--------A21 | A22
where, as we specified, A11 is a square banded matrix, A22 is a square dense matrix, and A21 and A12
are rectangular strips.
It is desirable to take advantage of the banded structure of A11. We can specify an algorithm for solving
a linear system A * x = b that can be written in terms of operations involving the sub-matrices A11,
A12, A21 and A22, which will achieve this goal, at the expense of a little extra work. One problem with
this technique is that it will fail if certain combinations of the matrices A11 and A22 are singular, which
can happen even when A is not singular.
The algorithm for solving A * x = b rewrites the system as:
A11 * X1 + A12 * X2 = B1
A21 * X1 + A22 * X2 = B2
The first equation can be solved for X1 in terms of X2:
X1 = - A11-1 * A12 * X2 + A11-1 * B1
allowing us to rewrite the second equation for X2:
( A22 - A21 * A11-1 * A12 ) X2 = B2 - A21 * A11-1 * B1
which can be solved as:
X2 = ( A22 - A21 * A11-1 * A12 )-1 * ( B2 - A21 * A11-1 * B1 )
The actual algorithm doesn't compute the inverse, of course, but rather factors the matrices A11 and A22
- A21 * A11-1 * A12.
Cartesian Basis Vectors

The Cartesian basis vectors are simply the N columns of the identity matrix, regarded as individual
column vectors.
These vectors form the standard basis for the set of vectors RN. The vector corresponding to the I-th
column of the identity matrix is often symbolized by ei.
Thus, if we are working in a space with dimension N of 4, the basis vectors e1 through e4 would be:
1
0
0
1
0
0
0
0
0
0
0
0
1
0
0
1
Facts about the Cartesian basis vectors:
The vectors have unit Euclidean norm, are pairwise orthogonal, and form a basis for the space of
N dimensional vectors;
A * ej yields a vector which is the J-th column of the matrix A;
the outer product of ei and ej a matrix A which is all zero except for the single entry A(I,J)=1.
The Cauchy-Schwarz Inequality

The Cauchy-Schwarz Inequality is a relationship between a vector inner product and a vector norm
derived from that inner product. In particular, if the norm ||*|| is defined by an inner product (*,*) as
follows:
|| x || = sqrt ( x, x ),
then the Cauchy-Schwarz inequality guarantees that for any vectors x and y it is the case that:
| ( x, y ) | <= || x || * || y ||.
The Cayley-Hamilton Theorem

The Cayley-Hamilton Theorem guarantees that every (square) matrix satisfies its own characteristic
equation.
For example, if A is the matrix:
2 3
1 4
then the characteristic equation is
lambda2 - 6 * lambda + 5 = 0.
which is not true for all values lambda, but just a few special values known as eigenvalues. The CayleyHamilton theorem guarantees that the matrix version of the characteristic equation, with A taking the
place of lambda, is guaranteed to be true:
A2 - 6 * A + 5 * I = 0.
CentroSymmetric Matrix
A centrosymmetric matrix is one which is symmetric about its center; that is,
Ai,j = Am+1-i,n+1-j
Example:
1 10
13 2
6 7
12 4
5 11
8 11 5
9 4 12
3 7 6
9 2 13
8 10 1
A centrosymmetric matrix A satisfies the following equation involving the Exchange matrix J:
J*A*J = A
.
Characteristic Equation
The characteristic equation of a (square) matrix A is the polynomial equation:
det ( A - lambda * I ) = 0
where lambda is an unknown scalar value.
The left hand side of the equation is known as the characteristic polynomial of the matrix. If A is of
order N, then there are N roots of the characteristic equation, possibly repeated, and possibly complex.
For example, if A is the matrix:
2 3
1 4
then the characteristic equation is

(
det (
2 - lambda
1
3
4 - lambda
)
)
= 0
or
lambda2 - 6 * lambda + 5 = 0.
This equation has roots lambda = 1 or 5.
Values of the scalar lambda which satisfy the characteristic equation are known as eigenvalues of the
matrix.
Some facts about the characteristic equation of A:
A and AT have the same characteristic equation.
The Cayley-Hamilton Theorem guarantees that the matrix itself also satisfies the matrix version of its
characteristic equation.
Cholesky Factorization
The Cholesky factorization of a positive semidefinite symmetric matrix A has the form:
A = L * LT
where L is a lower triangular matrix, or equivalently:

A = RT * R
where R is an upper triangular matrix.
If a matrix is symmetric, then it is possible to determine whether or not the matrix is positive definite
simply by trying to compute its Cholesky factorization: if the matrix has a zero eigenvalue, then it is
positive semidefinite, and the algorithm should theoretically spot this by computing a zero diagonal
element; if the matrix actually has a negative eigenvalue, then at a particular point in the algorithm, the
square root of a negative number will be computed.
Software to compute the Cholesky factorization often saves space by using symmetric matrix storage,
and overwriting the original matrix A by its Cholesky factor L.
As long as the matrix A is positive definite, the Cholesky factorization can be computed from an LU
factorization.
The Cholesky factorization can be used to compute the square root of the matrix.
The LINPACK routines SCHDC, SCHUD, SCHDD, SCHEX, SPOCO, SPOFA, SPODI, and SPOSL
compute and use the Cholesky factorization.
Circulant Matrix
A circulant matrix is a matrix whose first row of values is repeated in each successive row, shifted one
position to the right, with the end value "wrapping around".
Here is a square circulant matrix:
1
4
3
2
2
1
4
3
3
2
1
4
4
3
2
1
a "wide" rectangular circulant matrix:
1 2 3 4 5
5 1 2 3 4
4 5 1 2 3
a "tall" rectangular circulant matrix:

1
5
4
3
2
2
1
5
4
3
3
2
1
5
4
Simple facts about a (rectangular) circulant matrix A:
A is constant along any diagonal;

A is persymmetric;
A is a special kind of Toeplitz matrix;
Simple facts about a square circulant matrix A:
The Identity matrix is a circulant matrix;

A is normal, hence unitarily diagonalizable.
The product of two circulant matrices is a circulant matrix.
The inverse of a (nonsingular) circulant matrix is a circulant matrix.
Any two circulant matrices commute;
The transpose of a circulant matrix is a circulant matrix.
Every circulant matrix is diagonalized by the Fourier matrix.
The columns of the Fourier matrix are the eigenvectors of (every) the circulant matrix.
The vector (1,1,...,1) is an eigenvector of the matrix, with eigenvalue equal to the sum of the
entries of a row.
If W is an N-th root of unity, then
Y = A(1,1) + A(1,2)*W + A(1,3)*W2 + ... + A(1,N)*WN-1
is an eigenvalue of A, with (right) eigenvector:
( 1, W, W2, ..., WN-1 )
and left eigenvector:
( WN-1, WN-2, ..., W2, W, 1 ).

Although there are exactly Ndistinct N-th roots of unity, the circulant may have repeated
eigenvalues, because of the behavior of the polynomial. However, the matrix is guaranteed to
have Nlinearly independent eigenvectors.
Compare the concept of an anticirculant matrix.
Cofactor Matrix
The cofactor matrix of a square matrix A is generally used to define the adjoint matrix, or to represent
the determinant.
For a given matrix A, the cofactor matrix is the transpose of the adjoint matrix:
cofactor ( A ) = ( adjoint ( A ) )T
The determinant det(A) can be represented as the product of each of the entries of any given row or
column times their corresponding cofactor entries. In particular, consider the first row:
det(A) = A(1,1) * cofactor(A)(1,1) + A(1,2) * cofactor(A)(1,2) + ... + A(1,N) * cofactor
(A)(1,N)
The formula for the (I,J) entry of the cofactor matrix of A is:
cofactor(A)(I,J) = (-1)(I+J) * det ( M(A,I,J) )
where M(A,I,J) is the minor matrix of A, constructed by deleting row I and column J.
Column Echelon Form

Column echelon form is a special matrix structure which is usually arrived at by Gauss elimination.
Any matrix can be transformed into this form, using a series of elementary column operations. Once the
form is computed, it is easy to compute the determinant, inverse, the solution of linear systems (even for
underdetermined or overdetermined systems), the rank, and solutions to linear programming problems.
A matrix (whether square or rectangular) is in column echelon form if:
Each nonzero column of the matrix has a 1 as its first nonzero entry.
The leading 1 in a given column occurs in a row below the leading 1 in the previous column.
Columns that are completely zero occur last.
A matrix is in column reduced echelon form if it is in column echelon form, and it is also true that:
Each row containing a leading 1 has no other nonzero entries.
Column echelon form is primarily of use for teaching, and analysis of small problems, using exact
arithmetic. It is of little interest numerically, because very slight errors in numeric representation or
arithmetic can result in completely erroneous results.
Commuting Matrices
Two square matrices, A and B, are said to commute if
A*B=B*A
Facts about commuting matrices:
If a (real) matrix commutes with its transpose, it is a normal matrix.

Every (invertible) matrix commutes with its inverse.
Every circulant matrix commutes with all other circulant matrices of the same order.
Companion Matrix
The companion matrix for a monic polynomial P(X) of degree N is a matrix of order N whose
characteristic polynomial is P(X).
If the polynomial P(X) is represented as:
P(X) = XN + C(1) * X(N-1) + ... + C(N-1) * X + C(N).

then the companion matrix has the form:
0 0 0 ... 0 0 -C(N)
1 0 0 ... 0 0 -C(N-1)
0 1 0 ... 0 0 -C(N-2)
...................
0 0 0 ... 1 0 -C(2)
0 0 0 ... 0 1 -C(1)
Note that the characteristic polynomial, and hence the companion matrix, give a formula for expressing
the matrix AN in terms of lower powers of A.
Thus it is possible to construct a matrix with any desired set of eigenvalues, by constructing the
corresponding characteristic polynomial, and then the companion matrix.
The companion matrix can also be used to perform a decomposition of a matrix A. If x is a vector, and K
the Krylov matrix
K = Krylov ( A, x, n )
whose columns are the successive products x, A*x, A2*x, and so on, and if K is nonsingular, then
A = K * C * K-1
where the matrix C is the companion matrix of A.
(There are several equivalent forms of the companion matrix, with the coefficients running along the
top, the bottom, or the first column of the matrix.)
Compatible Norms
A matrix norm and a vector norm are compatible if it is true, for all vectors x and matrices A that
||A*x|| <= ||A|| * ||x||
In some texts, the word consistent is used in this sense, instead of compatible.
In particular, if you have not verified that a pair of norms are compatible, then the above inequality is
not guaranteed to hold. For any vector norm, it is possible to define at least one compatible matrix norm,
namely, the matrix norm defined by:
||A|| = supremum ||A*x|| / ||x||
where the supremum (roughly, the "maximum") is taken over all nonzero vectors x. If a matrix norm can
be derived from a vector norm in this way, it is termed a vector-bound matrix norm. Such a relationship
is stronger than is required by compatibility.
If a matrix norm is compatible with some vector norm, then it is also true that
||A*B|| <= ||A|| * ||B||
where both A and B are matrices.
Complex Number Representation

Complex numbers have the form a+bi, where i is a special quantity with the property that i2=-1.
It is possible to devise real matrices that behave like complex numbers. Let the value "1" be represented
by the identity matrix of order 2, and the value "i" be represented by
0
-1
1
0
Then it is easy to show that these matrices obey the rules of complex numbers. In particular, "i" * "i" = "1". In general, the complex number a+bi is represented by
a
-b
b
a
and multiplication and inversion have the correct properties.

Condition Number
The condition number of the coefficient matrix A of a linear system is a (nonnegative) number used to
estimate the amount by which small errors in the right hand side b, or in A itself, can change the solution
x.
This analysis ignores arithmetic roundoff, which is hard to analyze, and focusses on easily measurable
quantities known beforehand, and how they will amplify or diminish the roundoff errors.
Small values of the condition number suggest that the algorithm will not be sensitive to errors, but large
values indicate that small data or arithmetic errors may explode into enormous errors in the answer.
The condition number is defined in terms of a particular matrix norm. Many different matrix norms may
be chosen, and the actual value of the condition number will vary depending on the norm chosen.
However, the general rule that large condition numbers indicate sensitivity will hold true no matter what
norm is chosen.
The condition number for a matrix A is usually defined as
condition ( A ) = || A || * || A-1 ||.
If A is not invertible, the condition number is infinite.
Simple facts about the condition number:
The condition number is always at least 1;

The condition number of the identity matrix is 1;
The condition number of any orthogonal or unitary matrix is 1;
LINPACK routines such as SGECO return RCOND, an estimate of the reciprocal of the condition
number in the L1 matrix norm.
Turing's M condition number, M(A), for a matrix of order N, is defined as
M(A) = N * max | Ai,j | * max | A-1i,j |.
Turing's N condition number, N(A) is
N(A) = Frob ( A ) * Frob ( A-1 ) / N
where Frob(A) is the Frobenius matrix norm.

The Von Neumann and Goldstine P condition number is
P(A) = | lambda_Max / lambda_Min |
where lambda_Max and lambda_Min are the eigenvalues of largest and smallest magnitude, which is
equivalent to using the spectral radius of A and A-1.
There is also a condition number defined for the eigenvalue problem, which attempts to estimate the
amount of error to be expected when finding the eigenvalues of a matrix A.
Congruent Matrix
Congruent matrices A and B are related by a nonsingular matrix P such that
A = PT * B * P.
Congruent matrices have the same inertia.
Congruence is of little interest by itself, but the case where P is also an orthogonal matrix is much more
important.
Conjugate Gradient Method

The conjugate gradient method is designed to solve linear systems
A*x=b
when the matrix A is symmetric, and positive definite.
The method is not an iterative method, but rather a direct method, which produces an approximation to
the solution after N steps. Because of numerical inaccuracies and instabilities, many implementations of
the method repeat the computation several times, until the residual error is deemed small enough.
The method is ideally suited for use with large sparse systems, because the matrix A is only accessed to
compute a single matrix-vector product on each step. This involves no fill in or overwriting of the data
structure that describes A. However, if A is dense, the conjugate gradient method costs roughly 3 times
the number of operations for direct Gauss elimination.
The conjugate gradient method can be considered as a minimization of the functional f(x), defined by
f(x) = xT * ( 0.5 * A * x - b )
which achieves its minimum value when x solves the linear system.
Here are the formulas for the basic conjugate gradient method. Brackets indicate the value of an iterative
quantity. X[0] is the initial value of the vector X, X[1] the value after one iteration, and so on.
X[0] = 0
For K = 1 to N
Compute the residual error:

R[K-1] = B - A * X[K-1]
Compute the direction vector:

If K = 1 then
P[K] = R[0]
else
BETA = - PT[K-1] * A * R[K-1]
/ ( PT[K-1] * A * P[K-1] )
P[K] = R[K-1] + BETA * P[K-1]
end if
Compute the location of the next iterate:

ALPHA = ( RT[K-1] ) * R[K-1] / ( PT[K] * A * P[K] )
X[K] = X[K-1] + ALPHA * P[K]
end for
Conjugate gradient algorithms are available in IMSL, ITPACK and NSPCG.

Conjugate Matrix
The conjugate matrix of a complex matrix A, denoted by A* or conjugate ( A ), is the matrix obtained
by replacing each entry of A by its complex conjugate.
(In this document, the form conjugate ( A ) is preferred, because the A* is easily confused with
multiplication.
The complex conjugate transpose, sometimes called the Hermitian or tranjugate of A, is derived from A
by complex conjugation, followed by transposition, and is denoted by AH.
Conjugate of a Complex Number

The conjugate of a complex number z = a + b * i is the complex number
conjugate ( z ) = a - b * i.
The conjugate is frequently represented by placing a bar over the quantity, or occasionally a star after it,
as in z*.
The complex conjugate can be used in a formula for the norm or magnitude of a complex number, which
must always be a real nonnegative value:
norm ( z ) = sqrt ( z * conjugate ( z ) ) = sqrt ( a2 + b2 ).
For complex vectors, an inner product with the correct properties may be defined as:
V dot W = ( V, W ) = sum ( I = 1 to N ) conjugate ( V(I) ) * W(I).
This inner product is computed in the BLAS function CDOTC, for example, and yields another
relationship with the Euclidean vector norm:
|| V || = sqrt ( V dot V )
Conjunctive Matrix
Two (complex) matrices A and B are said to be conjunctive if there is some nonsingular matrix P so that
A = ( conjugate ( P ) )T * B * P,
This is the extension to complex matrices of the concept of a congruent real matrix.
Convergent Matrix
A convergent matrix A is a square matrix for which the limit as n goes to infinity of An is zero.
A matrix is convergent if and only if the spectral radius rho(A) satisfies
rho(A) < 1
A semiconvergent matrix A is a square matrix A for which the limit as n goes to infinity of An exists. If
a matrix is semiconvergent, it must be the case that
rho(A) <= 1
Cross Product
The cross product of two vectors u and v, denoted u x v, is a vector w which is perpendicular to u and v,
pointing in the direction so that (u,v,w) forms a right handed coordinate system, and whose length is
equal to the area of the parallelogram two of whose sides are u and v.
Algebraically,
w(1) = u(2) * v(3) - u(3) * v(2)
w(2) = u(3) * v(1) - u(1) * v(3)
w(3) = u(1) * v(2) - u(2) * v(1)
If the unit vectors in the coordinate directions are denoted by i, j and k, then the cross product vector can
also be regarded as the (vector) value of the following "determinant":
|
i
j
k |
w = u x v = det | u(1) u(2) u(3) |
| v(1) v(2) v(3) |
Cyclic Reduction
Cyclic reduction is a method for solving a linear system A*x=b in the special case where A is a
tridiagonal matrix.
On a parallel computer, this method solves the system in LOG(N) "steps" where N is the order of A. A
standard Gauss elimination method for a tridiagonal system would require roughly N "steps" instead.
A tridiagonal system has some very special properties that will allow us to carry this operation out.
Consider this system of 7 equations:
A11 x1 + A12 x2
=
A21 x1 + A22 x2 + A23 x3
=
A32 x2 + A33 x3 + A34 x4
=
A43 x3 + A44 x4 + A43 x5
=
A54 x4 + A55 x5 + A56 x6
=
A65 x5 + A66 x6 + A67 x7 =
A76 x6 + A77 x7 =
y1
y2
y3
y4
y5
y6
y7
The first equation can be used to eliminate the coefficient A21 in the second equation, and the third
equation to eliminate the coefficient A23 in the second equation. This knocks out variables x1 and x3 in
the second equation, but adds x4 into that equation.
By the same method, x3 and x5 can be eliminated from the equation for x4, and so on. By eliminating
the odd variables from the even equations, a smaller tridiagonal system system is derived, with half the
equations and variables.
If elimination is applied to this set, the number of equations is again reduced by half; this reduction may
be repeated until a single equation in one variable is reached. Backsubstitution then produces the values
of all the variables.
The reason this method might have an advantage over Gauss elimination is that, at each step of the
elimination phase, the parts of the step are independent. If many computer processors are available, then
each can be working on a separate portion of the elimination. If the number of processors is large
enough, the system can really be solved in LOG(N) time.
Cyclic reduction routines are available in the NCAR software library, the SLATEC library, and the Cray
SCILIB library.
Cyclic Tridiagonal Matrix

A cyclic tridiagonal matrix is a generalization of a tridiagonal matrix which includes an extra last entry
in the first row, and an extra first entry in the last row.
An example of a cyclic tridiagonal matrix:
-2 1 0 0 1
1 -2 1 0 0
0 1 -2 1 0
0 0 1 -2 1
1 0 0 1 -2
A cyclic tridiagonal matrix is not a tridiagonal matrix. If the matrix is constant along the three
generalized diagonals, a cyclic tridiagonal matrix is a circulant matrix. A cyclic tridiagonal matrix can
arise in situations where a periodic boundary condition is applied.
It is very disappointing that a cyclic tridiagonal matrix is not a tridiagonal matrix, since there are so
many good methods for solving tridiagonal linear systems. One way to solve a cyclic tridiagonal system
is to use the Sherman Morrison Formula and view the matrix as a rank one perturbation of a tridiagonal
matrix. Another was is to view it as a border banded matrix.
A cyclic tridiagonal matrix may also be called a periodic tridiagonal matrix.
Defective Matrix
A defective matrix is a (square) matrix that does not have a full set of N linearly independent
eigenvectors.
For every eigenvalue, there is always at least one eigenvector, and eigenvectors corresponding to distinct
eigenvalues are linearly independent. If the N eigenvalues of a matrix are distinct, then it surely has N
linearly independent eigenvectors, and so cannot be defective.
Conversely, if a matrix is defective, then it must have at least one repeated eigenvalue, that is, an
eigenvalue of algebraic multiplicity greater than 1. A matrix is defective if and only if its Jordan
Canonical Form has at least one nonzero entry on the superdiagonal.
Thus, a simple example of a defective matrix is:
1 1
0 1
which has the single eigenvalue of 1, with algebraic multiplicity 2, but geometric multiplicity 1. The
only eigenvector is ( 0, 1 ).
If a matrix is not defective, then its eigenvectors form a basis for the entire linear space. In other words,
any vector y can be written as
y=X*c
where X is the array of eigenvectors of A.
If a matrix A is not defective, then it is similar to its diagonal eigenvalue matrix:
A = X * LAMBDA * X-1
and the similarity transformation matrix X is actually the eigenvector matrix.
This in turn allows us to make interesting statements about the inverse, tranpose, and powers of A. For
instance, we see that
A2 = ( X * LAMBDA * X-1 ) * ( X * LAMBDA * X-1 )
= X * LAMBDA2 * X-1
leading us to the statement that for a nondefective matrix, the square has the same eigenvectors, and the
square of the eigenvalues.
Deflation
Deflation is a technique for "removing" a known eigenvalue from a matrix, in order to facilitate the
determination of other eigenvalues.
For example, the power method is able to estimate the eigenvalue of largest modulus of a matrix A.
Once this is computed, it might be desired to find the next largest eigenvalue. Deflation can be used to
essentially create a new matrix, A', which has the same eigenvalues as A, except that the largest
eigenvalue has been dropped (and the order of A' reduced by 1) or the largest eigenvalue is replaced by
0. In either case, the power method applied to A' will produce the next largest eigenvalue of A.
To eliminate a known eigenvalue, lambda, it is necessary to know its eigenvector x, which we will
assume has been scaled to have unit Euclidean norm. By the properties of eigenvectors, we know that
A * x = lambda * x.
Now define the matrix A' so that:
A' = A - lambda * x * xT.
Now x is an eigenvector of A' with eigenvalue 0, because:
A' * x = ( A - lambda * x * xT ) * x
= A * x - lambda * x * xT * x
= lambda * x - lambda * x
=0
If the power method is being employed, then the new iteration should try to "factor out" any component
of the eigenvector x; otherwise, small errors in the computation of the first eigenvalue and eigenvector
will interfere with the next results.
Theoretically, this process may be repeated as often as desired, eliminating each eigenvalue as it is
discovered. Practically, however, accumulated errors in the eigenvalues and eigenvectors make the
computation more and more unreliable with each step of deflation. Thus, if more than a few eigenvalues
are desired, it is more appropriate to use a standard technique.
Derogatory Matrix
A derogatory matrix is a matrix whose minimal polynomial is of lower degree than its characteristic
polynomial
Perhaps the only reason that the term is worth knowing is this fact: every nonderogatory matrix is
similar to the companion matrix of its characteristic polynomial.
Determinant of a Matrix
The determinant of a square matrix is a scalar value which is zero exactly when the matrix is singular.
If a matrix is singular, then it doesn't have an inverse and linear systems cannot be reliably solved. In
numerical work, the determinant is not a reliable indicator of singularity, and other data, such as the size
of the matrix elements encountered during pivoting, are preferred.
The determinant also occurs in the definition of the eigenvalue problem.
An explicit formula for the determinant of a matrix A is:
det ( A ) = sum [ over all P ] sign(P) * A(1,P(1)) * A(2,P(2) * ... * A(N,P(N)).
where the sum ranges over all possible permutations P of the numbers 1 through N, and sign(P) is +1 for
an even permutation, and -1 for an odd permutation. (Any permutation may be accomplished by a
sequence of switching pairs of objects. The permutation is called even or odd, depending on whether the
number of switches is even or odd).
A numerical method for finding the determinant comes as a byproduct of the LU factorization used in
Gaussian elimination. Typically, this factorization has the form
A = P * L * U,
and the value of the determinant is simply
det ( A ) = det ( P ) * product ( I = 1 to N ) U(I,I).
where det ( P ) is +1 or -1, again determined by the sign of the permutation.
Simple facts about the determinant:
the determinant of the identity matrix is 1;

the determinant of a permutation matrix is +1 or -1;
the determinant of an orthogonal matrix is +1 or -1;
the determinant of a diagonal matrix is the product of the diagonal entries;
the determinant of a unit upper or lower triangular matrix is 1;
the determinant of an upper or lower triangular matrix is the product of the diagonal entries;
the product of the eigenvalues of a matrix equals the determinant;
det ( AT ) = det ( A );
det ( A * B ) = det ( A ) * det ( B );
det ( A-1 ) = 1 / det ( A ).
A single elementary row operation has the following effect on the determinant:
Interchanging two rows multiplies the determinant by -1;

Multiplying a row by the nonzero scalar s multiplies the determinant by s;
Adding a multiple of one row to another leaves the determinant unchanged.
For small matrices, the exact determinant is simple to compute by hand. The determinant of a 2 by 2
matrix
a
c
b
d
is a*d-b*c, while the determinant of a 3 by 3 matrix:

a
e
h
b
f
i
c
g
j
is a * (f*j-g*i) - b * (e*j-g*h) + c * (e*i-f*h).

If absolutely necessary, the determinant of a matrix of order N can be computed recursively in terms of
determinants of minor matrices. Let M(A,I,J) stand for the (I,J) minor matrix of A. Then the
determinant of A is
det ( A ) = sum ( J = 1 to N ) (-1)(J+1) * A(I,J) * det ( M(A,I,J) ).
Of course, now we need to compute the determinants of the N minor matrices, but the order of these
matrices has been reduced by 1. Theoretically, we can represent the determinant of any of these matrices
of order N-1 by a similar sum involving minor matrices of order N-2, and this process can be repeated
until we reach matrices of order 1 or 2, whose determinants are easy to compute. In practice, this method
is never used except in simple classroom exercises.
There is a geometric interpretation of the determinant. If the rows or columns of A are regarded as
vectors in N dimensional space, then the determinant is the volume of a the parallelepiped, or "slanted
cube" whose one corner is defined by these vectors.
LAPACK and LINPACK provide routines for computing the determinant of a matrix, after the matrix
has been decomposed into LU factors.
Diagonal Dominance
A matrix is diagonally dominant if, for every column, the sum of the absolute values of the offdiagonal
elements is never greater than the absolute value of the diagonal element.
The matrix is strictly diagonally dominant if the offdiagonal sum is always strictly less than the absolute
value of the diagonal element.
The same definitions can be used to consider rows instead of columns. The terms column diagonally
dominant and row diagonally dominant may be used, if necessary, to specify which case is being
considered.
A strictly diagonally dominant matrix cannot be singular, by Gershgorin's Theorem.
A diagonally dominant matrix which is also irreducible cannot be singular.
Here is a diagonally dominant matrix which is not strictly diagonally dominant:
-2 1 0 0 0
1 -2 1 0 0
0 1 -2 1 0
0 0 1 -2 1
0 0 0 1 -2
For a linear system A * x = b, if the matrix A is strictly diagonally dominant, then both Jacobi iteration
and Gauss Seidel iteration are guaranteed to converge to the solution.

Diagonal Matrix
A diagonal matrix is one whose only nonzero entries are along the main diagonal. For example:
3 0 0
0 4 0
0 0 7
Simple facts about a diagonal matrix A:
A is singular if and only if any diagonal entry is zero;

The eigenvalues of A are the diagonal entries, and the eigenvectors are the columns of the
identity matrix;
The determinant of A is the product of the diagonal entries;
Diagonalizable Matrix
A diagonalizable matrix is any (square) matrix A which is similar to a diagonal matrix:
A = P * D * P-1.
This concept is important in the study of eigenvectors To see the relationship, post-multiply the equation
by P:
A * P = P * D.
Looking at the columns of P as eigenvectors, and the diagonal entries of D as eigenvalues, this shows
that a matrix is diagonalizable exactly when it has N linearly independent eigenvectors.
In certain cases, not only is a matrix diagonalizable, but the matrix P has a special form. The most
interesting case is that of any (real) symmetric matrix A; not only can such a matrix be diagonalized, but
the similarity matrix is orthogonal:
A = Q * D * Q-1 = Q * D * QT,
This fact can be interpreted to show that not only does every symmetric matrix have a complete set of
eigenvectors, but the eigenvectors and eigenvalues are real, and the eigenvectors are pairwise
orthogonal.
Similarly, a complex matrix that is defective.
If a matrix is normal, then it is not only diagonalizable, but the transformation matrix is unitary.
Downshift Matrix
The downshift matrix A circularly shifts all vector entries or matrix rows down 1 position.
Example:
0
1
0
0
0
0
1
0
0
0
0
1
1
0
0
0
Facts about the downshift matrix A:
A is a permutation matrix;
A is an N-th root of the identity matrix;
A is persymmetric;
A is a circulant matrix;
any circulant matrix, generated by a column vector v, can be regarded as the Krylov matrix (v,
A*v,A2*v,...,An-1*v.
the inverse of the downshift matrix is the upshift matrix.
Eigenvalues
Eigenvalues are special values associated with a (square) matrix, which can be used to analyze its
behavior in multiplying any vector.

The formal definition of an eigenvalue of a matrix A is that it is any value lambda which is a root of the
characteristic equation of the matrix,
det ( A - lambda * I ) = 0.
lambda is an eigenvalue of A if and only if there is a nonzero vector x, known as an eigenvector
(sometimes a "right" eigenvector), with the property that
A * x = lambda * x.
Note that there must also be a "left" eigenvector y, with the property
y * A = AT * y = lambda * y.
The characteristic equation has exactly N roots, so a matrix has N eigenvalues. An important
consideration is whether any eigenvalue is a repeated root, which determines how hard the eigenvector
computation will be.
If a matrix has the maximum possible number of linearly independent eigenvectors (namely N, the order
of the matrix), then the eigenvalues and eigenvectors can be used to diagonalize the matrix. This only
happens when the matrix is normal.
Simple facts about eigenvalues of A:
A is singular if and only if 0 is an eigenvalue of A.

If A is symmetric, all eigenvalues are real.
If A is positive definite symmetric matrix, all eigenvalues are real and positive.
Simple algorithms for computing eigenvalues include the power method and the inverse power method.
The QR method is a more powerful method that can handle complex and multiple eigenvalues.
LAPACK and EISPACK include algorithms for computing the eigenvalues and eigenvectors of a variety
of types of matrix, as well as methods that can be applied to more general eigenvalue problems.
Eigenvectors
A nonzero vector x is an eigenvector of the square matrix A if

A * x = lambda * x
for some scalar value lambda, called the associated eigenvalue.
Sometimes this eigenvector is more particularly described as a right eigenvector, so that we may also
consider left eigenvectors, that is, vectors y for which it is true that
y * A = AT * y = mu * y
for some scalar mu.
For every eigenvalue of a matrix, there is at least one eigenvector. Every nonzero multiple of this
eigenvector is also an eigenvector, but in an uninteresting way. If, and only if, an eigenvalue is a
repeated root, then there may be more than one linearly independent eigenvector associated with that
eigenvalue. In particular, if an eigenvalue is repeated 3 times, then there will be 1, 2 or 3 linearly
independent eigenvectors corresponding to that eigenvalue.
Facts about eigenvectors:
If x is an eigenvector, so is s*x, for any nonzero scalar s.

If A is singular, then it has an eigenvector associated with the eigenvalue 0, so that A * x = 0.
If x is a right eigenvector for eigenvalue lambda, and y is a left eigenvector for eigenvalue mu,
and lambda and mu are distinct, then x and y are orthogonal, that is, yT * x = 0. This property is
sometimes call biorthogonality.
EISPACK
EISPACK is a package of routines for handling the standard and generalized eigenvalue problems.
The beginning user who is not interested in trying to learn the details of EISPACK and simply wants the
answer to an eigenvalue problem quickly should call one of the main driver routines. Each of these is
tailored to handle a given problem completely with a single subroutine call. For more advanced work, it
may be worth investigating some of the underlying routines.
Driver routines to solve A*x=lambda*x include:
CG, complex general matrix.

CH, complex Hermitian matrix.
RG, real general matrix.
RS, real symmetric matrix.
RSB, real symmetric band matrix.
RSP, real symmetric, packed storage matrix.
RSPP, real symmetric packed storage matrix, some eigenvectors.
RST, real symmetric tridiagonal matrix.
RT, real sign-symmetric tridiagonal.
For the generalized eigenvalue problem:
RGG solves A*x=lambda*B*x for A and B real, general matrices.

RSG solves A*x=lambda*B*x for A real symmetric, B real positive definite.
RSGAB solves A*B*x=lambda*x for A real symmetric, B real positive definite.
RSGBA solves B*A*x=lambda*x for A real symmetric, B real positive definite.
EISPACK Matrix Norm

The EISPACK matrix norm is used in the EISPACK eigenvalue package.
The definition of the norm for an M by N matrix is:
||A|| = sum ( I = 1 to M, J = 1 to N ) | Ai,j |
It's a simple exercise to verify that this quantity satisifes the requirements for a matrix norm.
This norm is easy to calculate, and was used in EISPACK in order to have a standard against which to
compare the size of matrix elements that were being driven to zero. I haven't seen it used anywhere else
in practice.

Elementary column operations are a simple set of matrix operations that can be used to carry out Gauss
elimination, Gauss Jordan elimination, or the reduction of a matrix to column echelon form.
Restricting the operations to a simple set makes it easy to:
guarantee that each step is legitimate;

record each step using a simple notation;
compute the inverse of the total set of operations.
The three elementary column operations include:
interchange any two columns;

multiply any column by a nonzero value;
add a multiple of any column to another column.
Each of these operations may be represented by an elementary matrix, and the transformation of the
original matrix A to the reduced matrix B can be expressed as postmultiplication by a concatenation of
elementary matrices:
B = A * E(1) * E(2) * ... * E(k)
which may be abbreviated as:
B=A*C
Since C will be guaranteed to be invertible, we also know that,
B * C-1 = A
which yields a factorization of A.
Elementary Matrix
An elementary matrix E is one which, when pre-multiplying another matrix A, produces a product
matrix E * A which has exactly one of the following properties:
rows R1 and R2 are interchanged, or

row R1 is multiplied by a nonzero constant s, or
a multiple of row R2 is added to row R1.
The matrix E which interchanges rows R1 and R2 of matrix A has the form E(I,J)=:
1, for I = R1, J = R2, or I = R2, J = R1;

1, for I = J and I not equal to R1 or R2;
0, otherwise.
The inverse of this matrix is simply its transpose.

The matrix E which multiplies row R1 of A by the constant s has the form E(I,J)=:
s, if I = J = R1;
1, if I = J =/= R1;
0, otherwise.
The inverse of this matrix is constructed by negating the value of s.

The matrix E which adds s * row R2 to row R1 of A has the form E(I,J) =:
1, if I = J;
s, if I = R1 and J = R2;
0, otherwise.
The inverse of this matrix is constructed in the same way, using 1/s.
If a matrix F can be represented as the product of elementary matrices,
F = E1 * E2 * ... * EM,
then its inverse is:
F-1 = EM-1 * EM-1-1 * ... * E1-1.
An elementary similarity transformation uses a matrix F which is the product of elementary matrices,
and transforms the matrix A into the similar matrix B by the formula
B = F-1 * A * F.

Elementary row operations are a simple set of matrix operations that can be used to carry out Gauss
elimination, Gauss Jordan elimination, or the reduction of a matrix to row echelon form.
Restricting the operations to a simple set makes it easy to:
guarantee that each step is legitimate;

record each step using a simple notation;
compute the inverse of the total set of operations.
The three elementary row operations include:
interchange any two rows;

multiply any row by a nonzero value;
add a multiple of any row to another row.
Each of these operations may be represented by an elementary matrix, and the transformation of the
original matrix A to the reduced matrix B can be expressed as premultiplication by a concatenation of
elementary matrices:
B = E(k) * E(k-1) * ... * E(2) * E(1) * A
which may be abbreviated as:
B = C * A.
Since C will be guaranteed to be invertible, we also know that,
C-1 * B = A
which yields a factorization of A.
Ellipsoids
An ellipsoid is an N dimensional generalization of an ellipse. The formula for an ellipsoid may be
written as:
sum ( I = 1 to N, J = 1 to N ) A(I,J) * X(I) * X(J) = 1.
where A is a positive definite symmetric matrix.
A principal axis of an ellipsoid is any N dimensional point X on the ellipsoid such that the vector from
the origin to X is normal to the ellipsoid.
In the general case, there are exactly N principal axes (plus their negatives). In degenerate cases, there
may be an entire plane of vectors that satisfy the requirement, but it is always possible to choose a set of
N principal axes which are linearly independent.
Moreover, in the general case, the principal axes are pairwise orthogonal, and in the degenerate case,
may be chosen pairwise orthogonal.
Moreover, it is always true that the principal axes are eigenvectors of the matrix A of ellipsoid
coefficients. The length of the principal axis vector associated with an eigenvalue lambda(I) is 1 / Sqrt
( lambda(I) ).
These facts have a strong relationship to the formulation of the conjugate gradient method.
Equilibration
Equilibration is the technique of balancing the rows or columns of a matrix by rescaling them.
Consider, for instance, the fact that the following two equations are equivalent:
0.0001 * x + 0.0001 * y = 0.0001
and
1000 * x + 1000 * y = 1000
However, the large coefficients in the second equation will bias a Gauss elimination routine to choose
that equation as its pivot. Actually, it's more important in this case that the chosen row be as "linearly
independent as possible" from the other rows, and this is more likely to occur if we ensure that all the
rows start out with an equal norm. This can be done very simply, by finding the element of maximum
absolute value in each row and dividing that row (and its right hand side) by that value. Such a technique
is called row equilibration. It is not necessary that the rows have precisely the same norm; it is desirable
that the norms of the rows be maintained within some controlled range.
Equilibration is useful in many areas of linear algebra, including eigenvalue calculations. In some cases,
column equilibration is preferred, and in other cases, the norms of both the rows and columns are to be
controlled.
Equivalent Matrix
Matrices A and B are said to be equivalent if there are nonsingular matrices P and Q so that
A = P * B * Q.
Simple facts about equivalence:
Every matrix is equivalent to a diagonal matrix;

If A and B are equivalent, then they are both singular or both nonsingular.
Equivalence is a very loose concept of relatedness. A stronger and more useful concept is similarity.
Exchange Matrix
The exchange matrix J is constructed from an identity matrix by reversing the order of the columns.
For example, the matrix J of order 4:
0
0
0
1
0
0
1
0
0
1
0
0
1
0
0
0
Facts about the exchange matrix J:
J is a square root of the identity matrix;

J is a permutation.
J is symmetric.
J is persymmetric.
J is a Hankel matrix.
J is an anticirculant matrix.
The vector J*x comprises the entries of x in reverse order.
For any matrix A, the matrix J*A has the same rows but in reversed order.
For any matrix A, the matrix A*J has the same columns but in reversed order.
The exchange matrix is also called the anti-identity or counter-identity matrix.

External Storage Algorithms

An external storage algorithm is a method of solving a problem that is too large to be loaded into
computer memory as a whole.
Instead, the problem is solved incrementally, with most of the problem data residing, at any one time, in
computer files, also called "disk storage" or "external storage". It is a considerable difficulty just to
rewrite an algorithm that can handle a situation where, say, part of a matrix is in one place, and part is in
another, remote place.
However, such algorithms must also be aware that data transfers between memory and disk are very
slow. Hence, if the algorithm is to be of any use, it must do as much processing as possible on the
portion of the data that resides in memory, and read the external problem data into memory as rarely as
possible, and in large contiguous "chunks".
Fourier Matrix
The Fourier matrix represents the linear operator that transforms a vector of data into a vector of Fourier
coefficients.
Let w indicate an N-th root of unity. Then, if we choose N=4, the matrix F will be:
1
1
1
1
1
w
w^2
w^3
1
w^2
w^4
w^6
1
w^3
w^6
w^9
which simplifies to
1
1
1
1
w
w^2
w^3
w^2
1
w^2
w^3
w^2
w^1
However, we will choose to scale F by 1/sqrt(N).

Facts about the Fourier matrix F:
F is symmetric.
F is unitary;
F is a fourth-root of the identity matrix.
the only eigenvalues of F are 1, -1, i and -i.
F diagonalizes any circulant matrix.
F is a special kind of Vandermonde matrix.
Frobenius Matrix Norm

The Frobenius matrix norm is a matrix norm that has the simple formula: ||A|| = the square root of the
sum of the squares of all the entries of the matrix.
The Frobenius matrix norm is not a vector-bound matrix norm, although it is compatible with the L2
vector norm, and much easier to compute that the L2 matrix norm.
The Frobenius matrix norm is sometimes called the Schur matrix norm or Euclidean matrix norm
Gauss Elimination
Gauss elimination has the goal of producing a solution x to the system of linear equations A*x=b, where
A is matrix of order N, and b a vector of length N. The standard version of Gauss elimination used in
most algorithms employs partial pivoting.
Gauss elimination accomplishes its goal by decomposing the original matrix A into three factors, a
permutation matrix P, a unit lower triangular matrix L, and an upper triangular matrix U. The factors are
related to the original matrix by the formula
A = P * L * U.
Once the matrix is factored, it is a simple matter to solve A*x=b, by solving instead P * ( L * ( U *
x ) ) ) = b, because each of the three factors is easy to invert.
Moreover, once the factors are known, the user may solve several linear systems involving A, with
different right hand sides.
The determinant of A is equal to the product of the determinants of the factors, and hence is easily
computed: the determinant of P is plus or minus 1, and that of L is 1, and that of U is simply the product
of its diagonal elements.
The inverse matrix could be solved for, if necessary, by solving the N linear systems A * X(I) = E(I),
where E(I) is the I-th Cartesian basis vector The vectors X(1), X(2), ..., X(N) then are the columns of the
inverse of A.
As an example of the Gauss elimination of a matrix, suppose we start with with the the matrix:
A
1 2 3
4 5 6
7 8 0
We can write an imperfect PLU factorization as:

P
1 0 0
0 1 0
0 0 1
1 0 0
0 1 0
0 0 1
1 2 3
4 5 6
7 8 0
The factorization is imperfect because, although A = P*L*U, the matrix U is not upper triangular. We
will now modify the matrix U, and update the factors P and L, so that it is always true that A=P*L*U,
while the matrix U gradually is transformed into the correct upper triangular form.
Step 1.1: Choose a pivot row in U, namely, row 3. We want to interchange rows 3 and 1 of the matrix U.
The elementary permutation matrix P(1,3) does this. We are allowed to insert the inverse of this matrix
times itself between L and U in the factorization. We also insert the inverse of this matrix times itself
between P and L. If we use primes to denote the updated quantities, these operations are:
A = P * L * U
= [ P * P-1(1,3) ] * [ P(1,3) * L * P-1(1,3) ] * [ P(1,3) * U ]
= P' * L' * U'
The resulting factorization is now:

P
0 0 1
0 1 0
1 0 0
1 0 0
0 1 0
0 0 1
7 8 0
4 5 6
1 2 3
Step 1.2: Eliminate U2,1 by subtracting 4/7 of row 1 from row 2. To do this, we construct the elementary
matrix L(1,2,4/7), and insert the product of its inverse and itself into the factorization. Then we absorb
the inverse into L, and the matrix into U.
A = P * L * U
= P * [ L * L-1(1,2,4/7) ] * [ L(1,2,4/7) * U ]
= P * L' * U'

P
0 0 1
0 1 0
1 0 0
L
1 0 0
4/7 1 0
0 0 1
U
7 8 0
0 3/7 6
1 2 3
A = P * L * U
= P * [ L * L-1(1,3,1/7) ] * [ L(1,3,1/7) * U ]
= P * L' * U'

P
0 0 1
0 1 0
1 0 0
L
1 0 0
4/7 1 0
1/7 0 1
U
7 8 0
0 3/7 6
0 6/7 3
Step 2.2: Choose a pivot row in U, namely, row 3. We want to interchange rows 3 and 2 of the U. The
elementary permutation matrix P(2,3) does this. We are allowed to insert the inverse of this matrix times
itself between L and U in the factorization. We also insert the inverse of this matrix times itself between
P and L. If we use primes to denote the updated quantities, these operations are:
A = P * L * U
= [ P * P-1(2,3) ] * [ P(2,3) * L * P-1(2,3) ] * [ P(2,3) * U ]
= P' * L' * U'

P
0 1 0
0 0 1
1 0 0
L
1 0 0
1/7 1 0
4/7 0 1
U
7 8 0
0 6/7 3
0 3/7 6
A = P * L * U
= P * [ L * L-1(2,3,1/2) ] * [ L(2,3,1/2) * U ]
= P * L' * U'

P
0 1 0
0 0 1
1 0 0
L
1
0 0
1/7 1 0
4/7 1/2 1
U
7 8
0
0 6/7 3
0 0 9/2
The PLU factorization is now correct.

You should be able to see a formula for the final factors:
P = I * P-1(1,3) * P-1(2,3)
and
L = P(2,3) * P(1,3) * I * P-1(1,3) * L-1(1,2,4/7) * L-1(1,3,1/7) * P-1(2,3) * L-1(2,3,1/2)
and
U = L(2,3,1/2) * P(2,3) * L(1,3,1/7) * L(1,2,4/7) * P(1,3) * A
You should see that the form of these matrices guarantees that A=P*L*U. The way we carried out the
steps guarantees that P stays a permutation matrix, L stays a lower triangular matrix, and U becomes an
upper triangular matrix.
Gauss Jordan Elimination

Gauss Jordan elimination is a method for solving a system of linear equations A * x = b for x, or for
computing the inverse matrix of A.
Gauss elimination and Gauss Jordan elimination are very closely related. Gauss elimination reduces A to
an upper triangular matrix, and saves the elimination factors in a lower triangular matrix. Gauss Jordan
elimination proceeds relentlessly until A has been converted into the identity matrix.
Thus, unlike the Gauss elimination procedure, the Gauss Jordan elimination does not produce a
factorization of the matrix, but only a solution to the linear system. This means that if a second linear
system has to be solved, the matrix has to be set up and eliminated all over again.
The simplest way to describe Gauss Jordan is to note that to solve, say, the linear system A * x = b, the
right hand side is appended as an extra column of the matrix. Then, on step I of the elimination, we
choose a pivot row, move it to row I, divide it through by the pivot value, and then eliminate the matrix
entries in column I from all other rows, rather than simply from rows I+1 through N. When the process
is completed, the solution x has overwritten the right hand side b that was stored in column N+1.
Several right hand sides can be handled at once, by appending all of them to the coefficient matrix; the
inverse can be computed by appending a copy of the identity matrix to the coefficient matrix before
beginning elimination.
Gauss Jordan elimination is primarily used as a teaching tool, and for small linear systems. In practical
computation, standard Gauss elimination is universally preferred.
Gauss Seidel Iteration For Linear Equations

The Gauss Seidel iteration for linear equations is an iterative method for solving linear systems of
equations A*x=b. It is similar to the the Jacobi algorithm and the successive overrelaxation method
(SOR).
The Gauss Seidel iteration should only be used for matrices which are symmetric and positive definite,
or for a matrix which is strictly diagonally dominant.
Each step of the Gauss Seidel iteration begins with an approximate answer x, and produces a new
approximation y. Each component of y is computed in order, using the formula:
y(i) = [ b(i)
- a(i,1)*y(1)
- a(i,2)*y(2)
...
- a(i,i-1)*y(i-1)
- a(i,i+1)*x(i+1)
...
- a(i,n)*x(n) ] / a(i,i)
The calculation of each entry of y is dependent on the calculation of the entries of lower index. Thus, the
value of y(1) is calculated first, and then y(2) is calculated based on the values of y(1) as well as x(3)
through x(n), and so on.
The process is to be repeated until the residual error is small, or the change in the approximate solution
is negligible.
The Gauss Seidel iteration can be considered in terms of its matrix splitting. That is, if we decompose
the matrix A into its strictly lower triangular, diagonal, and strictly upper triangular parts:
A=L+D+U
then the method is equivalent to the iteration
( L + D ) * xnew = b - U * x.
which means that the convergence of the algorithm can be understood in terms of the behavior of powers
of the iteration matrix:
- ( L + D )-1 * U,
which in turn may best be understood by looking at the eigenvalues.
If the original coefficient matrix A is symmetric, then it may be preferred to use the symmetric Gauss
Seidel iteration or SGS. In this case, the iteration consists of pairs of Gauss Seidel steps. The odd steps
are the same as the usual iteration. But in the even steps, the variables are solved for in reverse order.
Each pair of such steps is a single step of the SGS iteration, which has the property that its iteration
matrix is similar to a symmetric matrix (though not necessarily symmetric itself). Among other things,
this means that SGS can be used as a preconditioner for certain other problems.

A general matrix is one which has no special matrix structure or matrix symmetry.
In such a case, there are no space advantages to be gained by using a special matrix storage format, and
so the matrix entries are stored using the standard two dimensional array format provided by the
programming language.
For general matrices, the only remaining issue concerns the problem that occurs when the matrix storage
must be set aside before the size of the matrix is known. In FORTRAN, for example, it is common to
specify a maximum matrix size of, say, 100 by 100. If the actual problem to be solved is of size 25 by
25, then it may be necessary to describe the data with both the matrix order of 25, and the leading
dimension of the storage array, which is 100. In LINPACK and LAPACK, variables containing leading
dimension information have names like LDA, LDB and so on.
LAPACK and LINPACK provide routines, with the prefix SGE, which apply to matrices in general
storage.
Generalized Permutation Matrix

A generalized permutation matrix is a square matrix A with at most one nonzero entry in each row, and
in each column.
A standard permutation matrix, of course, has exactly one nonzero entry in each row and column, and
that entry has value 1.
An interesting fact: if A is a nonsingular nonnegative matrix, then the inverse of A is also nonnegative if
and only if A is a generalized permutation matrix. In other words, it's very hard for both A and A-1 to be
nonnegative, and essentially can only happen if A is diagonal.
So suppose A >= 0 but A is not a generalized permutation matrix. For an arbitrary vector b >= 0 , can
we say that the solution x of A * x = b is nonnegative? No, because we know that the inverse of A is not
nonnegative. Therefore A-1 contains at least one negative entry, say entry (i,j). Choose b = E(J), where E
(J) is the J-th Cartesian basis vector. Then x = A-1 * b and it is easy to see that x(i) is negative.
Gershgorin Disks
The method of Gershgorin disks provides an estimate of the size of the eigenvalues of a matrix. The
accuracy of the estimate varies wildly, depending on the size of the elements of the matrix. It is most
useful for matrices that are diagonally dominant or sparse.
Gershgorin's theorem states that the eigenvalues of any matrix A lie in the space covered by the disks D
(I):
D(I) = ( x: sqrt ( x - A(I,I) )2 <= R(I) )
where R(I) is the sum of the absolute values of the off-diagonal elements of row I:
R(I) = sum ( J =/= I ) | A(I,J) |.
The theorem may also be applied using columns instead of rows.
Givens Rotation Matrix

A Givens rotation is a linear transformation applied to two vectors, or two rows or columns of a matrix,
which can be interpreted as a coordinate axis rotation. The intent of the rotation is to zero out an entry of
the vector or matrix using an orthogonal transformation.
A Givens rotation is similar to the elementary row operation that adds a multiple of one row to another,
but because a Givens rotation is an orthogonal similarity transformation, it offers greater stability and
easy invertibility.
A Givens rotation matrix G has the form:
1 0
0 c
0 0
0 0
0 -s
0 0
^
col i
0
0
1
0
0
0
0
0
0
1
0
0
0
s
0
0
c
0
0
0
0
0
0
1
<-- row i
<-- row j
^
col j
where c = cosine(theta) and s = sin(theta) for some angle theta.

Premultiplying A by G has the following effect:
row i
row j
of A is replaced by
of A is replaced by
c*row i + s*row j
-s*row i + c*row j
in G*A;
in G*A.
while postmultiplication, A*G, would carry out a similar operation on the columns of A.
As an example, to zero out entry A(I,J) of a matrix requires a Givens rotation with values of cosine and
sine so that:
- s * A(I,I) + c * A(I,J) = 0.
It's not actually necessary to compute the underlying rotation angle theta, since c and s can be computed
directly:
s = A(I,J) / sqrt ( A(I,J)2 + A(I,I)2 )
c = A(I,I) / sqrt ( A(I,J)2 + A(I,I)2 )
For instance, to zero out the 3,1 entry of this matrix:
4 2 0
0 4 5
3 8 1
the sine and cosine are 3/5, 4/5, yielding a Givens matrix G of:
0.8
0
-0.6
0
1
0
0.6
0
0.8
and the product G * A:

5.0 6.4 6.0
0
4
5
0 5.2 8.0
It is possible to zero out entries of a matrix, one by one, using Givens rotations, similar to the way that
Householder matrices are used, to reduce a matrix to a simpler form. The process can be used to zero out
the entire lower triangle of a matrix, but further operations on the upper triangle would reintroduce
zeroes in the lower triangle. Nonetheless, zeroing out the lower triangle means that Givens rotations can
be used to produce the QR factorization of the matrix.
Gram Schmidt Orthogonalization

Gram Schmidt orthogonalization is a process which starts with a set of N vectors X(I), each with M
components, and produces a set of N2 orthonormal vectors Y which span the linear space of the original
vectors.
N2 is less than N if the vectors X(I) are linearly dependent, and equal to N if they are linearly
independent. Thus, one use for the Gram Schmidt process is simply to determine if a set of vectors are
linearly dependent; a second is to determine the dimension of the space spanned by a linearly dependent
set.
The Gram Schmidt process may be defined iteratively:
for I = 1 to N
N2 = I
Y(I) = X(I)
for J = 1 to I-1
C(J) = dot_product ( X(I), Y(J) )
Y(I) = Y(I) - C(J) * Y(J)
end for
Norm = sqrt ( dot_product ( Y(I), Y(I) ) )
if ( Norm = 0 ) exit
Y(I) = Y(I) / Norm
end for
Another way of looking at the process is to use the vectors to form the columns of a matrix A. The Gram
Schmidt process can then be used to construct one version of the QR factorization of the matrix A:
A=Q*R
where Q is orthogonal and R is upper triangular.
Hadamard Product
The Hadamard product of matrices A and B is a matrix C created by elementwise multiplication:
Ci,j = Ai,j * Bi,j
The Hadamard product is defined for any pair of rectangular matrices, as long as they have the same
"shape", that is, the same number of rows, and the same number of columns.
Example of a Hadamard product:
1 2 3
4 5 6
7 8 9 = 7 16 27
10 11 12
40 55 72
Hadamard's Inequality
Hadamard's inequality provides an upper bound on the size of the determinant of a matrix. It is related
to the fact that the determinant represents the volume of an N-dimensional parallelepiped.
Let ||C(I)|| designate the Euclidean norm of column I of the matrix A. Hadamard's inequality states that
det ( A ) <= ||C(1)|| * ||C(2)|| * ... * ||C(N)||,
with equality holding only if one of the C(I)'s is zero, (yielding the minimum possible value of 0), or if
all the C(I)'s are orthogonal (yielding the largest possible value).
The theorem may also be applied using rows instead of columns.
Hankel Matrix
A Hankel matrix is a matrix which is constant along each of its anti-diagonals.
Here is an example of a square Hankel matrix:

7
6
5
4
6
5
4
3
5
4
3
2
4
3
2
1
and a rectangular Hankel matrix:

7 6 5 4 3 2
6 5 4 3 2 1
5 4 3 2 1 0
Simple facts about a Hankel matrix A:
A is symmetric;
the inverse of A is symmetric, but need not be a Hankel matrix;
Compare the concepts of Toeplitz Matrix, an Anticirculant Matrix, and a Persymmetric Matrix.
Harwell Boeing Sparse Matrix Collection (HBSMC)

The Harwell Boeing Sparse Matrix Collection, or HBSMC, is a set of 43 data files describing a standard
set of test sparse matrices for sparse matrix calculations.
The test set comprises linear systems, least squares problems, and eigenvalue calculations from a wide
variety of disciplines. The set is offered as a standard benchmark for comparison of algorithms.
Here is an overview of the source and size of the various matrices:
Discipline
Counter examples, small matrices

Original Harwell test set
Air traffic control
Number
of
matrices
3
36
1
Largest
order
Largest number
of nonzeroes
11
822
2873
76
4841
15032
Astrophysics
Chemical Engineering
Circuit simulation
Demography
Economic modelling
Nuclear reactor core modelling
Optimal power flow problems
Stochastic modelling
Acoustic scattering
Oil reservoir modelling
Stiff ODE problems
George and Liu test problems
Model PDE problems
Navier Stokes problems
Unassembled finite element matrices
Oceanography
Power network matrices
Everstine test set, ship structures
Structures, eigenproblems
Structures, linear equations
Least squares problems
2
16
1
3
11
3
3
7
4
19
10
21
3
7
10
4
14
30
22
36
4
765
2021
991
3140
2529
1374
4929
1107
841
5005
760
3466
900
3937
5976
1919
5300
2680
15439
44609
1850
24382
7353
6027
543162
90158
8606
47369
5664
4089
20033
5976
13681
4322
25407
15680
17159
13571
23853
133840
1029655
10608
The SPARSKIT package includes utilities for conversion of matrices in the Harwell Boeing format into
other formats, such as that used by ELLPACK and ITPACK.

The Harwell Boeing Sparse Matrix Collection uses a matrix storage format which is a special kind of
sparse matrix storage for most of the matrices in the collection.
The standard sparse matrix format is column oriented. That is, the matrix is represented by a sequence of
columns. Each column is held as a sparse vector, represented by a list of row indices of the entries in an
integer array and a list of the corresponding values in a separate real array. A single integer array and a
single real array are used to store the row indices and the values, respectively, for all the columns.
Data for each column are stored in consecutive locations. The columns are stored in order, and there is
no space between columns. A separate integer array holds the location of the first entry of each column,
and the first free location. For symmetric and Hermitian matrices, only the entries of the lower triangle
are stored, including the diagonal. For antisymmetric matrices, only the strict lower triangle is stored.
Here is a simple example of a 5 by 5 matrix:

1.0 -3.0 0.0 -1.0
0.0 0.0 -2.0 0.0
2.0 0.0 0.0 0.0
0.0 4.0 0.0 -4.0
5.0 0.0 -5.0 0.0
0.0
3.0
0.0
0.0
6.0
This matrix would be stored in the arrays
COLPTR (location of the first entry of a column),

ROWIND (row indices) and
VALUES (numerical values)
as follows:
Subscripts:
COLPTR
ROWIND
VALUES
1
1
1.0
4
3
2.0
6
8
5
1
5.0 -3.0
10
11
10
12
4
2
5
1
4
4.0 -2.0 -5.0 -1.0 -4.0
2
3.0
5
6.0
We can generate column 5, say, by observing that its first entry is in position COLPTR(5)=10 of arrays
ROWIND and VALUES. This entry is in row ROWIND(10)=2 and has value VALUES(10)=3.0. Other
entries in column 5 are found by scanning ROWIND and VALUES to position COLPTR(6)-1, that is,
position 11. Thus, the only other entry in column 5 is in row ROWIND(11)=5 with value VALUES(11)
=6.

The HBSMC finite element storage format is a special matrix storage format for those matrices in the
collection which derive from a finite element problem.
Matrices arising in finite element applications are usually assembled from numerous small elemental
matrices. The collection includes a few sparse matrices in original unassembled form. The storage of the
individual unassembled matrices is based on the general sparse format, which stores a matrix as a list of
matrix columns. The elemental representation stores the matrix as a list of elemental matrices. Each
elemental matrix is represented by a list of the row/column indices (variables) associated with the
element and by a small dense matrix giving the numerical values by columns, or in the symmetric case,
only the lower triangular part. The lists of indices are held contiguously, just as for the lists of row
indices in the standard format. The dense matrices are held contiguously in a separate array, with each
matrix held by columns. Although there is not a one to one correspondence between the arrays of integer
and numerical values, the representation does not hold the pointers to the beginning of the real values for
each element. These pointers can be created from the index start pointers (ELTPTR) after noting that an
element with NU variables has NU*NU real values, or (NU*(NU+1))/2 in the symmetric case.
We illustrate the elemental storage scheme with a small, 5 by 5 example:
5.0
0.0
0.0
1.0
2.0
0.0
4.0
3.0
0.0
6.0
0.0
3.0
7.0
8.0
1.0
1.0 2.0
0.0 6.0
8.0 1.0
9.0 0.0
0.0 10.0
generated from four elemental matrices:

1
4
1 (2.0 1.0)
4 (1.0 7.0)
1
5
1 (3.0 2.0)
5 (2.0 8.0)
2
3
5
2 (4.0 3.0 6.0)
3 (3.0 5.0 1.0)
5 (6.0 1.0 2.0)
3
4
3 (2.0 8.0)
4 (8.0 2.0)
where the variable indices are indicated by the integers marking the rows and columns. This matrix
would be stored in the ELTPTR (location of first entry), VARIND (variable indices) and VALUES
(numerical values) arrays as follows:
14
8.
Subscripts:
15
ELTPTR:
VARIND:
VALUES:
2.
1
1
2.
3
4
1.
5
1
7.
8 10
5
2
3. 2.
3
8.
5
4.
3
3.
4
6.
10
5.
11
1.
12
2.
13
2.
Hermite Normal Form

A nonsingular integer matrix is in Hermite Normal Form if it is lower triangular, all entries are nonnegative, and each row has a unique maximum element which is located on the main diagonal.
(In some definitions, the matrix is required to be upper triangular instead.)
Any nonsingular integer matrix can be transformed to Hermite Normal Form using a series of
unimodular column operations:
add an integer multiple of one column to another;

exchange two columns;
multiple any column by -1.
For example, given the matrix A:

5
-4
0
2
2
-3
1
4
6
its Hermite normal form is:

1
4
6
0
6
15
0
0
30
Hermitian Matrix
A Hermitian matrix A is a complex matrix that is equal to its complex conjugate transpose:
A = AH
Here is a Hermitian matrix:
1
1-2i
3+4i
1+2i
4
-6i
3-4i
6i
8
Simple facts about a Hermitian matrix A:
The diagonal entries of A must be real.

The eigenvalues of A are real, although the eigenvectors are generally complex.
Eigenvectors of A that correspond to different eigenvectors will be orthogonal.
A can be decomposed into the form A = U*LAMBDA*UH where U is unitary and LAMBDA is
real and diagonal.
The corresponding concept for real matrices is symmetric.

Hessenberg Matrix
An upper Hessenberg matrix is a matrix which is entirely zero below the first subdiagonal.
An upper Hessenberg matrix is "almost" upper triangular. A lower Hessenberg matrix is, of course,
entirely zero above the first superdiagonal. Upper Hessenberg matrices occur so often that they are
frequently simply called "Hessenberg" matrices.
An example of an upper Hessenberg matrix is:
1
6
0
0
0
2
7
5
0
0
3
8
4
8
0
4
9
3
3
9
5
1
2
7
1
Eigenvalue programs typically transform a matrix into upper Hessenberg form, and then carry out the
QR method on this matrix, which converges rapidly to a matrix which is diagonal except for 2 by 2
blocks corresponding to complex eigenvalues.
The reason for transforming a matrix into upper Hessenberg form is that the QR method is much less
expensive if carried out on an upper Hessenberg matrix. It is actually cheaper to go to the additional
trouble of transforming a matrix to upper Hessenberg form, and then carrying out the QR method on
that matrix, rather than carrying out the QR method on the original matrix.
Householder Matrix
A Householder matrix for a given vector v has the form:
H = I - 2 * v * vT / ( norm2 ( v ) )2
or, in the common case where the Euclidean norm of v is 1, we may write:
H = I - 2 * v * vT
For the simple case where w = (1/3, 2/3, 2/3), here is what H would look like:
(1 0 0)
(1/9 2/9 2/9)
( 7/9 -4/9 -4/9)
(0 1 0) - 2 * (2/9 4/9 4/9) = (-4/9 1/9 -8/9)
(0 0 1)
(2/9 4/9 4/9)
(-4/9 -8/9 1/9)
A little "reflection" will convince you that the Householder matrix for any vector v will always be
symmetric, just like this example.
The Householder matrix is also orthogonal:
H * HT = H * H = I.
Householder matrices can be used to compute the QR factorization of a matrix. A Householder matrix
can be found which will "wipe out" all the subdiagonal entries of the first column of the original matrix.
Another Householder matrix can be found which will "wipe out" all the subdiagonal entries of the
second column, and so on. At the end of N-1 steps of this process, we have computed
Hn-1 * ... * H2 * H1 * A = R
where R is upper triangular. But the product
H = Hn-1 * ... * H2 * H1
is an orthogonal matrix. We can multiply both sides by its transpose, which is also its inverse, to get
A = HT * R
or, if we define Q to be HT:

A = Q * R.
The Householder matrix is sometimes called an elementary reflector.
Idempotent Matrix
An idempotent matrix A has the property that
A*A=A
An idempotent matrix is sometimes called a projector matrix.
Simple facts about an idempotent matrix A:
The identity matrix is idempotent;

Every point of the form A*x is a fixed point of A, that is A * ( A * x ) = A * x;
If A is idempotent, and it is not the identity matrix, it must be singular;
The only eigenvalues of A are 0 and 1;
I-A is also idempotent;
rank(A) = trace(A);
I-2*A is involutory;
The Identity Matrix

The identity matrix, usually denoted I, is a square matrix with 1's on the main diagonal and 0's
elsewhere. The identity matrix behaves like the number 1 in matrix multiplication. For any matrix A:
A * I = I * A = A.
Here is the 3 by 3 identity matrix:
1 0 0
0 1 0
0 0 1
Simple facts about the identity matrix I:
The norm of I must be 1, for any vector-bound matrix norm

The condition number of I must be 1, for any vector-bound matrix norm;
IT = I-1 = I;
The eigenvalues of I are 1, with algebraic and geometric multiplicity N;
Every nonzero vector x is an eigenvector of I. A basis for the eigenspace of I is the set of
columns of I.
I is diagonal;
I is idempotent;
I is involutory;
I is symmetric;
I is an orthogonal projector;
Ill Conditioned Linear System

When solving a linear system A * x = b, the matrix A is said to be ill conditioned if small errors or
perturbations in the coefficient matrix A or right hand side b correspond to large errors or perturbations
in the solution x.
A numerical scale for ill conditioning is provided by the condition number.
Linear systems which are extremely ill conditioned may be impossible to solve accurately. A standard
example of an ill conditioned matrix is the Hilbert Matrix, with A(I,J) = 1 / ( I + J ).
The Incomplete LU factorization

The incomplete LU factorization is an approximate LU factorization of a matrix A, comprising a unit
lower triangular matrix L and an upper triangular matrix U.
Typically, the matrix A is large and sparse, and an iterative scheme is being used to solve the linear
system A*x=b. The incomplete LU factorization is intended as a preconditioner, which modifies the
linear system, improving the convergence rate of the iterative scheme. In particular, the preconditioner
matrix is M=L*U, and we are well advised to store L and U instead of M, since this gives us an easy
way of solving linear systems associated with M.

The computation of the incomplete LU factorization is similar to that of the usual PLU factors except
for two points. First, we assume that no pivoting is required, hence the P factor can be omitted.
Secondly, if the original matrix A has a zero entry, then we require that the corresponding entry of the U
matrix be zero. This means that there is no fill in (and it is also why L*U will not equal A). Because
there is no fill in, the factors L and U can actually be stored in a data structure that is the same as that of
A. The elements of L are stored in the positions devoted to lower triangular elements of A, while U goes
in the diagonal and upper triangular locations.
Inertia
The oddly named inertia of a (square) matrix is the numbers of negative, zero, and positive eigenvalues.
Sylvester's Law of Inertia states that if A and B are congruent matrices, then we cannot guarantee that
they have the same eigenvalues, but they do have the same inertias.
This theorem allows us to determine if a symmetric matrix is positive definite. Because the matrix is
symmetric, we can compute an LDL factorization:
A = L * D * LT
where L is unit lower triangular, and D is diagonal. This means that A is congruent to the diagonal
matrix D. But the eigenvalues of D are easily determined, from which we can get the inertia of D. This is
equal to the inertia of A; in particular, if D has only positive eigenvalues, then so does A, which is
therefore positive definite.
Moreover, if such a factorization is cheap, as for a tridiagonal symmetric matrix, then we can search for
eigenvalues by seeking diagonal shifts of the matrix that cause the number of negative eigenvalues to
change by 1.
Inner Product
An inner product is a scalar-valued function of two vectors x and y, denoted (x,y), with the properties
that:
( x, y ) = ( y, x ), symmetry
( x, s * y ) = s * ( x, y ) for any scalar s, linearity
( x, y + z ) = ( x, y ) + ( x, z ) additivity
( x, x ) >= 0, and equal to zero only if x = 0, positivity.
( x, A * y ) = ( AT * x, y ).
A vector inner product (x,y) can be used to define a corresponding vector norm ||x||:
|| x || = sqrt ( x, x ).
If the inner product and norm are related in this way, then the Cauchy-Schwarz inequality relates them.
The inner product is sometimes referred to as the dot product (because it is often represented as x dot y),
or as the scalar product (because its result is a scalar value).
Inverse Matrix
The inverse matrix of a square matrix A, if it exists, is a matrix denoted A-1 with the property that
A * A-1 = A-1 * A = I.
If the inverse matrix exists, it is unique, and A is said to be nonsingular or invertible. Otherwise, A is
singular.
If the inverse of A exists, then the solution of
A*x=b
can be immediately written down:
x = A-1 * b.
However, it's not a good idea to solve a linear system in this way. The inverse is relatively expensive to
compute, and subject to greater inaccuracies than other solution methods. This is not to say that the
inverse isn't useful. Orthogonal and unitary transformations are so popular in numerical linear algebra
because their inverses "come for free"; (and their inverses are very well conditioned).
Simple facts:
if lambda is an eigenvalue of A, then 1/lambda is an eigenvalue of A-1;

if x is an eigenvector of A, x is also an eigenvector of A-1.
LAPACK and LINPACK include routines for explicitly computing the inverse of a given matrix.
Inverse Power Method

The inverse power method is a technique for solving the eigenvalue problem.
The inverse power method is related to the power method, but is more flexible. Through the use of
shifts, it can be "aimed" to seek the eigenvalue that is closest to any particular target value. Moreover,
once a rough estimate of an eigenvalue is made, the shift can be set to this value, which will increase the
convergence rate of the algorithm.
The inverse power method only requires that the user be able to repeatedly solve linear systems of the
form:
A*x=b
or, when shifts are used,
( A - shift * I ) * x = b.
Thus, for instance, if A is a band matrix or a sparse matrix, the user can employ a storage method and
solution algorithm appropriate to the particular form of the problem.
The inverse power method gets its name from the fact that, when the shift is zero, it is equivalent to
using the power method on the matrix A-1.
The inverse power method begins by picking a starting guess for the eigenvector x, and the eigenvalue
lambda. If the shift will not vary then a fixed value should be set now. Then repeat the following steps
as often as necessary:
Set norm = the norm of x;

Set x = x / norm;
Set lambda_Old = lambda;
Set lambda = 1 / norm;
If the shift may vary, set shift = lambda;
Solve ( A - shift * I ) * xnew = x.

If no convergence, set xnew = x and repeat.
Invertible Matrix
An invertible matrix A is a (square) matrix for which there exists an inverse matrix B, called the inverse
of A.
A square matrix A is invertible if and only if:
no eigenvalue of A is zero;
no singular value of A is zero;
every linear system A * x = b has a unique solution;
the null space of A is exactly the zero vector;
the rows (and the columns) of A are linearly independent;
the determinant of A is nonzero;
Gauss elimination, with partial pivoting, and exact arithmetic, never encounters a zero pivot
(including the "unused" pivot on the last step).
Involutory Matrix
An involutory matrix A has the property that
A * A = I.
Simple facts about an involutory matrix A:
The identity matrix is involutory;

A is invertible, and A-1=A;
Irreducible Matrix
An irreducible matrix is a (square) matrix which is not reducible.
Definition One: A reducible matrix is one which can be rearranged into the following block form:
( P Q )
( 0 R )
where P and R are square sub-blocks, and "0" represents a (nonempty) rectangular block of zero entries.
The rearrangement can be done by simultaneous row and column permutations. An irreducible matrix is
one which is not reducible.
Definition Two: A matrix is irreducible if and only if, for any row index i and column index j, there is
always a nonnegative integer p (which may be 0) and a sequence of integers k1, ..., kp so that the
product
Ai,k1 * Ak1,k2 * ... * Akp,j
is nonzero.
Definition Three: A nonnegative matrix A is irreducible if and only if, for any vector x>0 it is the case
that A*x>0.
Definition Two implies that if the entries of A are nonnegative, then all entries of AN-1 are positive
except for the diagonal entries which may be zero. (This has implications for the use of the power
method.)
Definition Three: A matrix of order N is irreducible if, for any division of the integers between 1 and N
into two disjoint sets K1 and K2, there is always a nonzero element A(I,J) with I in K1 and J in K2.
The concept of an irreducible matrix is mainly of interest in the analysis of the convergence of certain
iterative schemes for linear equations. One key idea is the following: if the matrix A is irreducible and
diagonally dominant, then A is nonsingular.
If you only know that the matrix is diagonally dominant, then Gershgorin's theorem would still not rule
out an eigenvalue of zero, and hence singularity. It's the irreducibility that guarantees no zero eigenvalue
here. The "1, -2, 1" tridiagonal matrix is an example where this theorem applies.
Here's an example of an irreducible matrix with zero main diagonal:
0 1 0 0
1 0 1 0
0 1 0 1
0 0 1 0
If you're familiar with graph theory, then a matrix A is irreducible if and only if the digraph with
corresponding adjacency matrix is strongly connected. (Consider node I to be connected to node J if A(I,
J) is nonzero. The digraph is strongly connected if you can get from any node to any other node
following directed edges.)
Iterative Methods for Eigenvalues

A method is called iterative when it consists of a basic series of operations which are carried out over
and over again, until the answer that is produced is no longer significantly changing, or some
exceptional error occurs, or the limit on the number of steps is exceeded.
All eigenvalue problems are solved by iterative methods, except for the "toy" problems presented in
textbooks. This is because the computation of eigenvalues is equivalent to finding the roots of a
polynomial, and there is no explicit method for finding the roots of a general polynomial of degree 5 or
higher.
The best known methods for the eigenvalue problem include:
the power method;

the inverse power method;
the Jacobi Algorithm;
the QR method.
Iterative Methods for Linear Equations

A method is called iterative when it consists of a basic series of operations which are carried out over
and over again, until the answer that is produced is no longer significantly changing, or some
exceptional error occurs, or the limit on the number of steps is exceeded.
A direct method is the "opposite" of an iterative method. A fixed number of operations are carried out
once, at the end of which the solution is produced. Gauss elimination on a linear system is an example of
such a direct method. Direct methods are the primary method for solving small or dense or
nonsymmetric linear systems.
The most common reason for using an iterative method is that it can require far less storage than a direct
method. An iterative method typically only sets aside storage for the original nonzero entries of the
matrix; no fill in occurs. Standard direct methods must set aside an entry for every possible position in
the matrix, though some reduction in this requirement is possible if the matrix is banded.
Secondly, each iteration typically takes much less time than a full direct solve; thus, it is possible, for
some problems, that an iterative method will actually converge to an acceptable answer more quickly
than a direct method.
An iterative method has numerous disadvantages. You will need a starting point, and if you pick a poor
one that may slow down convergence. Your system matrix usually needs to satisfy extra conditions
beyond merely being nonsingular. The rate of convergence may be extremely slow, although this can be
helped by a suitable preconditioner. If you have a very large problem and are storing the matrix in a
compact form, the programming and computational cost involved in storing and retrieving coefficient
data can exceed that of the solution phase.
Iterative methods are generally only suitable for certain kinds of system matrices. The most common
requirements are that the matrix be positive definite symmetric or strictly diagonally dominant.
Iterative methods for solving systems of linear equations include:
Gauss Seidel Iteration;;

the Jacobi Algorithm ;
the conjugate gradient method;
Successive Overrelaxation Method (SOR).
Iterative Refinement
Iterative refinement is an attempt to "improve" a computed solution x0 for the linear system of equations
A*x=b.
The algorithm is occasionally effective when the coefficient matrix is ill conditioned, a problem that
may become evident if the residual error is computed, and seen to be relatively large.
A single step of the algorithm involves computing the residual error:
r := b - A * x,
solving the linear system:

A * dx = r,
and adding the correction dx to the original solution x.
x := x + dx.
In order to achieve any improvement in accuracy, the residual error should be calculated in higher
arithmetic precision. The rest of the calculation can be done in single precision, allowing the use of the
already computed LU factorization of A, if available. However, to compute the residual error, the
original matrix A is needed. It is preferable to figure out how to use the LU factors to compute the
residual error, rather than keeping two copies of the matrix, one factored and one untouched.
If the residual error of the new solution is still too large, but somewhat better, then the procedure may be
repeated as often as desired.
A subroutine SGEIM for carrying out iterative refinement using the LINPACK routines is described in
the LINPACK User's Guide.
Jacobi Algorithm for Eigenvalues

The Jacobi algorithm for computing eigenvalues is an iterative method, which may be applied to a
symmetric matrix A, to compute its eigenvalues and eigenvectors.
The method will produce an orthogonal matrix Q and an "approximately" diagonal matrix lambda such
that
A = QT * lambda * Q.
The steps of the iteration compute a sequence of orthogonal matrices Q1, Q2, ... which transform the
original matrix A into matrices A1, A2, ... each of which has the same eigenvalues as A, but each of
which is "more" diagonal than A was. Starting with A, we determine Q1, and produce A1, which is
orthogonally similar to A:
A1 = Q1T * A * Q1.
We then determine a matrix Q2 that produces A2:
A2 = Q2T * A1 * Q2
= ( Q1 * Q2 )T * A * ( Q1 * Q2 ).
The point of the algorithm is how we choose the matrices Q at each step. Q1 is chosen in such a way as
to "annihilate" the (1,2) and (2,1) elements of A. That is, the (1,2) and (2,1) elements of A1 will be zero.
Q2 will eliminate the (1,3) and (3,1) elements of A1. Thus A2 will have zeroes in the (1,3) and (3,1)
positions. Unfortunately, the (1,2) and (2,1) positions of A2, which we just zeroed out on the previous
step, will not remain zero, but will fill in again! However, the fill in values are generally smaller than the
original values. As we can sweep through the entire matrix, we repeatedly annihilate the off-diagonal
elements until they all have decreased below some tolerance.
It can be shown that the sum of the squares of the off diagonal elements always decreases to zero with
the iteration. Thus, for some iteration step M, one can expect to have a matrix AM and a matrix
Q = Q1*Q2*...*QM
so that
AM = QT * A * Q
where AM is "essentially" diagonal. At that point, we can rewrite this equation as
A * Q = lambda * Q
where lambda is the diagonal entries of AM, and is the eigenvalues of A, and the transformation matrix
Q is the matrix of eigenvectors of A. Thus, if the off diagonal elements disappear as promised, we have
approximately solved our eigenvalue problem.
So which orthogonal matrix Q zeroes out a specific pair of entries? The formula is fairly simple. To
annihilate the arbitrary entries A(I,J) and A(J,I), the matrix Q is equal to the identity matrix, except for:
Q(I,I) =
Q(J,I) =
C
- S
Q(I,J) = S
Q(J,J) = C
where C and S are the cosine and sine of some rotation angle THETA. Thus, each matrix Q is a Givens
rotation matrix. We can compute C and S directly (skipping the computation of THETA) by the
following formula:
U
T
C
S
=
=
=
=
( A(J,J)
sign ( U
1 / sqrt
T / sqrt
)
(
(
A(I,I) ) / ( 2 * A(I,J) )
/ ( | U | + sqrt ( U2 + 1 ) )
T2 + 1 )
T2 + 1 )
The Jacobi method is simple and easy to program, but is usually slower than the QR method.
The Jacobi Algorithm for Linear Equations

The Jacobi algorithm for linear equations is an iterative method for solving linear systems of equations
A * x = b.
The method is similar to the Gauss Seidel and Successive Overrelation Method (SOR), but has the
distinction of being the only simple iterative method that is easy to program in parallel.
The Jacobi iteration is only appropriate for matrices which are strictly diagonally dominant or else
symmetric and positive definite.
Each step of the Jacobi iteration begins with an approximate solution x. An improved approximate
solution xnew is computed by solving N independent linear equations:
xnew(i) = [ b(i)
- A(i,1)
* x(1)
- A(i,2)
* x(2)
...
- A(i,i-1) * x(i-1)
- 0.0
* x(i)
<-- note that we are skipping x(i) here!
- A(i,i+1) * x(i+1)
...
- A(i,n)
* x(n) ] / A(i,i)
Once xnew is computed, the residual error is calculated:

r = A * xnew - b
and the solution increment:

dx = xnew - x.
If the norms of these vectors are satisfactorily small, the iteration may be halted. Otherwise, xnew
becomes the new starting guess x for the next step of the iteration.
The Jacobi iteration can be considered in terms of its matrix splitting. That is, if we decompose the
matrix A into its strictly lower triangular, diagonal, and strictly upper triangular parts:
A=L+D+U
D * xnew = b - ( L + U ) * x.
- D-1 * ( L + U ),
which in turn may best be understood by looking at the eigenvalues. Note that if A is symmetric, then so
is the iteration matrix.
The Jacobi Preconditioner

The Jacobi preconditioner is a very simple preconditioning matrix for use with an iterative method for
linear equations.
For a given system matrix A, the Jacobi preconditioner matrix M is the diagonal matrix whose entries
are defined by
M = diag ( A )
In effect, using the Jacobi preconditioner amounts to dividing each equation and right hand side entry by
the corresponding diagonal entry of the original coefficient matrix.
Jordan Canonical Form

The Jordan canonical form of a matrix A is an upper bidiagonal matrix whose main diagonal contains
the eigenvalues of A, and whose superdiagonal contains only zeroes or ones. The location of the ones in
the superdiagonal is determined by the algebraic and geometric multiplicity of the eigenvalues.
Here is an example of a Jordan canonical form:
4
0
0
0
0
0
2
0
0
0
0
1
2
0
0
0
0
1
2
0
0
0
0
0
3
Every matrix is unitarily similar to its Jordan Canonical Form. That is, for any matrix A, there exists a
unitary matrix U so that
A = U* * J * U
where J has Jordan canonical form. This form can also be regarded as a matrix factorization.
If A is real, but has complex eigenvalues, the matrix J has complex entries.
The Jordan canonical form is of little interest to computational linear algebraists. Unless exact arithmetic
is used, it is extremely sensitive to small errors, and the information it provides can be computed more
reliably in other ways.
L1 Matrix Norm
The L1 matrix norm is a matrix norm that is vector-bound to, and hence compatible with, the L1 vector
norm.
Thus, the formal definition of the norm is
||A|| = max ( || A*x || / ||x|| )
where the vector norm used on the right hand side is the L1 vector norm, and the maximum is taken over
all nonzero vectors x.
However, it is easy to show that the L1 matrix norm has a simpler formula: ||A|| = the maximum, over
all matrix columns, of the sum of the absolute values of the entries in the column.
L1 Vector Norm
The L1 vector norm is a vector norm defined as
||x|| = sum ( 1 <= I <= N ) |x(i)|.
L2 Matrix Norm
The L2 matrix norm is a matrix norm that is vector-bound to, and hence compatible with, the L2 vector
norm.
||A|| = max ( || A*x || / ||x|| )
where the vector norm used on the right hand side is the L2 vector norm, and the maximum is taken over
all nonzero vectors x.
The L2 matrix norm has another formulation: ||A|| = the square root of the maximum absolute value of
the eigenvalues of AT * A.
The computation of the L2 norm is expensive, and so it is often simpler to use the easily-computed
Frobenius matrix norm, which is not vector-bound to the L2 vector norm, but is compatible with it.
L2 Vector Norm
The L2 vector norm is a vector norm defined as
||x|| = sqrt ( sum ( 1 <= I <= N ) x(i)2 )

The L2 vector norm is also known as the Euclidean vector norm or the root-mean-square vector norm.
L Infinity Matrix Norm

The L Infinity matrix norm is a matrix norm that is vector-bound to, and hence compatible with, the L
Infinity vector norm.
||A|| = max ( || A*x || / ||x|| )
where the vector norm used on the right hand side is the L Infinity vector norm, and the maximum is
taken over all nonzero vectors x.
However, it is easy to show that the L Infinity matrix norm has a simpler formula: ||A|| = the maximum,
over all matrix rows, of the sum of the absolute values of the entries in the row.
L Infinity Vector Norm

The L Infinity vector norm is a vector norm defined as
||x|| = max ( 1 <= I <= N ) |x(i)|.
LAPACK
LAPACK is a set of linear algebra routines, intended as the replacement for LINPACK and EISPACK.
It is a project sponsored by the Argonne National Laboratories and the Numerical Algorithms Group
(NAG). The LAPACK routines are intended to achieve optimal performance on various machines by
calling the level 3 BLAS, which use block methods to achieve high performance.
LDL Factorization
The LDL factorization of a symmetric matrix is a decomposition of the form:
A = L * D * LT,
involving a unit lower triangular matrix L, and a diagonal matrix.
The LDL factorization is a special case of the LU Factorization, in which we give up the option of
Pivoting in order to get a very simple factorization. If the matrix A has a zero pivot, the factorization is
still valid; we just can't guarantee that we can solve linear systems.
If the matrix A is actually positive definite, then we can get the even stronger Cholesky factorization.
Linear Dependence
A set of M vectors, each of order N, is called linearly dependent if there is some linear combination of
the vectors, with at least one nonzero coefficient, which equals the zero vector.
If the I-th vector is used as row I of an M by N matrix A, then this is equivalent to saying there is a
nonzero vector C such that
A*C=0
If no such combination is possible, then the vectors are linearly independent.
Simple facts:
if any of the vectors is the zero vector, then the set is linearly dependent;
if M is greater than N, the set is linearly dependent;
if M = N, the vectors are linearly independent if and only if the matrix A is nonsingular.
Linear Least Squares Problems

A typical linear least squares problem seeks to solve a linear system A * x = b where the matrix A is
"unsuitable" for treatment by the standard methods used for square, nonsingular, well-conditioned
coefficient matrices.
In the most common case, there are more equations than unknowns, so that A is actually a matrix with
rectangular order of M rows by N columns. The right hand side may or may not be consistent, and A
itself may perhaps not have the fullest possible rank.
Despite the fact that this problem can't be solved by the usual means, it is still the case that:
a unique exact solution x may exist;

many exact solutions might exist;
there might be no exact solutions, but clearly some vectors x are "better" than others, in the sense
that they produce a smaller residual error b - A * x.
The solution of a linear least squares problem, then, is partly to determine which of the above three cases
is occurring. Moreover:
If there is a unique exact solution, it should be obtained.

If there are many exact solutions, then they form a linear space, and that linear space should be
described;
If there are no exact solutions, then a "best" solution, which produces the smallest possible
residual error, should be produced.
One approach to the problem is to figure out a way that will allow standard techniques to be applied. A
square coefficient matrix can be constructed by replacing the M by N rectangular system
A*x=b
by the square system of order N:
AT * A * x = AT * b.
This linear system is known as the normal equations. If the columns of the original matrix A are
independent, then AT * A is invertible, and the system can be solved by Gauss elimination. The answer
x will not usually satisfy any of the original equations exactly, but it will be the answer that minimizes
the Euclidean norm of the residual error.
We can use the normal equations to guide us in how the answer can be gotten, at least for systems with
maximal rank. However, the coefficient matrix of the normal equations is usually ill conditioned (its
condition number being the square of the condition number of A). Other methods of solving this
problem are preferred, usually via the QR factorization or the pseudoinverse. Such methods can also
handle the case where the matrix does not have maximal rank.
Linear Space
A linear space is a collection X of "vectors", a scalar field F, (usually the real or complex field), and the
operations of vector addition and scalar multiplication, with the properties that:
X includes the zero vector;

if x is in X, then so is alpha * x, for any scalar alpha;
if x and y are in X, then so is x + y.
Examples of linear spaces include:
The set R^n of N dimensional vectors;

the null space of a matrix;
the eigenspace associated with an eigenvalue of a matrix, the set of all eigenvectors associated
with a particular eigenvalue of a matrix (plus the 0 vector, which we usually don't count as an
eigenvector);
the set of all vectors perpendicular to a given vector.
A linear space has a dimension. The dimension of a linear space can be thought of as the smallest
number of vectors from which the space can be reconstructed, or the cardinality of the smallest set of
vectors that spans the space, or the cardinality of any basis for the space.
A Linear Transformation is, formally, an operator A applied to elements x of some linear space X, with
the properties that
A ( r * x ) = r * A ( x ) for any x in X, and any real number r;

A ( x1 + x2 ) = A ( x1 ) + A ( x2 ) for any x1 and x2 in X;
If X is a finite dimensional vector space with an orthonormal basis, then any element x can be
represented by an N dimensional vector, and any linear transformation on X can be represented by a
matrix. It is usually this matrix that people think of when they speak of a linear transformation.
Linear transformations are frequently used, for example, in computer graphics. It is interesting to note
that for an arbitrary linear transformation matrix A, the Gram Schmidt factorization allows us to write A
as the product of an orthogonal matrix Q and an upper triangular matrix R. If we "factor out" the
diagonal entries of R, we then can view A as:
A=Q*D*S
where
Q is orthogonal, and represents a set of rotations;

D is diagonal, and represents a set of dilations;
S is unit upper triangular, and represents a set of shears.
LINPACK
LINPACK is a standard linear algebra package for factoring matrices, computing matrix determinants,
condition numbers, and inverses, and for solving linear systems. Additional capabilities include least
squares solutions, QR and singular value decomposition.
Many matrix storage modes are allowed, including dense, banded, symmetric, positive definite,
symmetric banded, but LINPACK does not handle sparse matrices, nor does it employ iterative methods.
Here are the routines available for single precision computations. There is a related set available for
complex matrices, with names which begin with C instead of S.
SCHDC computes the Cholesky Factorization of a positive definite matrix in general storage.
SCHDD "downdates" a Cholesky Factorization.
SCHEX updates the Cholesky Factorization of a permuted matrix.
SCHUD updates a Cholesky Factorization.
SGBCO factors a general band matrix and estimates its condition number.
SGBDI computes the determinant of a matrix factored by SGBCO or SGBFA.
SGBFA factors a general band matrix.
SGBSL solves a linear system involving a general band matrix.
SGECO factors a general matrix and estimates its condition number.
SGEDI gets determinant or inverse of a matrix factored by SGECO or SGEFA.
SGEFA factors a general matrix.
SGESL solves a linear system factored by SGECO or SGEFA.
SGTSL solves a linear system, unfactored tridiagonal matrix.
SPBCO factors a positive definite band matrix and estimates its condition number.
SPBDI computes the determinant or inverse of a matrix factored by SPBCO or SPBFA.
SPBFA factors a positive definite band matrix.
SPBSL solves a linear system factored by SPBCO or SPBFA.
SPOCO factors a positive definite matrix and estimates its condition number.
SPODI computes the determinant or inverse of a matrix factored by SPOCO or SPOFA.
SPOFA factors a positive definite matrix.
SPOSL solves a linear system factored by SPOCO or SPOFA.
SPPCO factors a positive definite packed matrix and estimates its condition number.
SPPDI computes the determinant or inverse of a matrix factored by SPPCO or SPPFA.
SPPFA factors a positive definite packed matrix.
SPPSL solves a linear system factored by SPPCO or SPPFA.
SPTSL solves a linear system for a positive definite tridiagonal matrix.
SQRDC computes the QR factorization of a general matrix.
SQRSL solves an overdetermined system, given QR factorization.
SSICO factors a symmetric indefinite matrix and estimates its condition number.
SSIDI computes the determinant or inverse of a matrix factored by SSICO or SSIFA.
SSIFA factors a symmetric indefinite matrix.
SSISL solves a linear system factored by SSIFA or SSICO.
SSPCO factors a symmetric indefinite packed matrix and estimates its condition number.
SSPDI computes the determinant or inverse of a matrix factored by SSPCO or SSPFA.
SSPFA factors a symmetric indefinite packed matrix.
SSPSL solves a linear system factored by SSPFA or SSPCO.
SSVDC computes the singular value decomposition of a general matrix.
STRCO estimates the condition number of a triangular matrix.
STRDI finds the inverse or determinant of a triangular matrix.
STRSL solves a triangular linear system.
LU Factorization
The LU factorization of a matrix is a decomposition of the form:
A = P * L * U,
involving a permutation matrix P, a unit lower triangular matrix L, and an upper triangular matrix U.
P records the pivoting operations carried out in Gauss elimination, L the row multipliers used during
elimination and U contains the pivot values and other information.
The factors are typically computed via Gauss elimination. Once the factors are computed, they may be
used to
solve one or more linear systems A*x=b;

solve one or more transposed linear systems AT*x=b;
compute the inverse;
compute the determinant.
The LU factorization is generally computed only for a nonsingular square matrix, but the LU
factorization exists and is useful even if the matrix is singular, or rectangular.
While the LU factorization is usually quite satisfactory, it is possible to prefer a factorization of the form
A = P * L * U * Q,
by using Gaussian elimination with complete pivoting, in which case Q is a permutation matrix selecting
the variable (or column) to be eliminated at each step. Another rival factorization is the QR factorization
A = Q *R,
which is slightly more expensive, but has better stability and accuracy properties.
LAPACK and LINPACK include routines to compute the LU factorization of a matrix stored in a
variety of formats. Note that the P, L and U factors themselves are scrambled and compressed in the
data storage used by these routines, so that it's difficult to determine their actual values.
M Matrix
An M matrix is a (real) (square) invertible matrix whose offdiagonal elements are nonpositive, and
whose inverse is a nonnegative matrix.
An M matrix is also called a Minkowski matrix.
There are many definitions of an M matrix. Another one is that a matrix A is an M matrix if there exists
a nonnegative matrix B, with a maximal eigenvalue r, such that
A=c*I-B
where c >= r. From this definition, it should be clear that an M matrix must have a nonnegative
diagonal, and nonpositive offdiagonal.
Facts about an M matrix A:
All of the eigenvalues of A have nonnegative real part; (and if a matrix has nonpositive
offdiagonal elements, and all eigenvalues have nonnegative real part, it is an M matrix);
Every real eigenvalue of A is nonnegative;
Every principal submatrix of Ais an M matrix;
Every principal minor matrix of A is nonnegative;
The inverse of A is a nonnegative matrix.
A symmetric M matrix is called a Stieltjes matrix. A Stieltjes matrix is positive definite.

Magic Square
A magic square can be regarded as a square matrix with an associated constant mu such that all row and
column sums are equal to mu. In most cases the main and antidiagonal elements also sum to mu.
Most magic squares of order n are made from the consecutive integers from 1 to n2. In this case, it is
common that an entry k and its "complement" n+1-k are located symmetrically with respect to the
center of the matrix. Such a magic square is called regular.
Example:
1 14 15 4
12 7 6 9
8 11 10 5
13 2 3 16
When a magic square is regarded as a matrix, it may exhibit some interesting properties. For instance, if
we let J be the Exchange matrix, then
A + J * A * J = 2 * ( mu / n ) * E
where E is the matrix all of whose entries are 1.
Because the entries of A are positive, the Perron Frobenius theorem guarantees that there is a positive
eigenvalue, equal to the spectral radius of the matrix, and which is a simple eigenvalue (having algebraic
multiplicity of 1), and an associated eigenvector all of whose entries are positive.
In fact, the dominant simple eigenvalue is mu, and the associated positive eigenvector is e=(1,1,...,1)T.
Matrix Exponential
The matrix exponential of a square matrix A is a matrix B(A,t) = exp ( A * t ), which has properties
similar to those of the exponential function of a scalar argument.
In particular:
B(A,0) = I, the identity matrix;

d B(A,t)/dt = A * B(A,t);
Lambda is an eigenvalue of A if and only if exp ( Lambda * t ) is an eigenvalue of B(A,t).
B(A,t) is never singular, for any values of A or t;
B(A,t) = sum ( I = 0 to Infinity ) ( A * t )I / I!;
Inverse ( B(A,t) ) = B(A,-t);
If A = M * J * Inverse(M) is the Jordan canonical form of A, then B(A,t) = M * exp ( J*t ) *
Inverse(M). If J is diagonal, then exp(J*t) is a diagonal matrix whose entries are the
exponentials of J(I,I)*t;
If the matrix A can be is orthogonally diagonalizable, so that

A = Q * Lambda * QT,
then, using the power series definition of B(A,t) results in:
B(A,t) = Q * sum ( I = 0 to Infinity ) ( Lambda * t )I / I! * QT;
or
B(A,t) = Q * exp ( Lambda, t ) * QT,
where exp ( Lambda, t ) is the diagonal matrix whose I-th diagonal entry is exp(lambda(i)), the
exponential of the I-th eigenvalue of A.
In the general case where A cannot be orthogonally diagonalized, or where the eigenvalues and
eigenvectors cannot be reliably computed, the computation of the matrix exponential is difficult.
Matrix Factorization
Matrix factorization is the process of rewriting a matrix A as a product of factors with certain special
properties. Matrix factorization is a key technique in solving linear systems, determining eigenvalues,
and many other tasks.
Useful matrix factorizations include:
The Cholesky Factorization

Incomplete LU Factorization
Jordan_Canonical_Form
The LDL Factorization
The LU Factorization
The QR Factorization
Row Echelon Form
The Singular Value Decomposition
Matrix multiplication is the computation of the matrix-vector product A * x or the matrix-matrix product
A * B, where A and B are (possibly rectangular) matrices, and x is a vector.
A matrix product is only meaningful when the factors are conformable. This is a condition on the
dimensions of the factors. If we are computing A * x or A * B, then the column order of A must equal
the order of x or the row order of B.
To multiply the L by M array A times the M by N array B, the product C will have L rows and N
columns, and a typical entry C(I,J) is:
C(I,J) = sum ( K = 1 to M ) A(I,K) * B(K,J).
Matrix multiplication is not commutative. A*B and B*A are not even of the same order unless both
matrices are square. Even if they are square, the two products will general have different numerical
values.
As a computational task, matrix multiplication is relatively expensive. To multiply two matrices of order
N takes roughly 2*N3 floating point operations, which is more expensive than Gauss elimination of a
matrix of order N. On vector and parallel machines, it is important to write a multiplication algorithm
carefully, to take advantage of the potential for speedup.
Matrix Norm
A matrix norm is a scalar quantity, which may be thought of as a sort of "magnitude" of the matrix. The
norm can be used to estimate the effect of multiplying the matrix times a vector, solving a linear system,
or other matrix operations. The norm also is used in the analysis of error and convergence
A matrix norm ||*|| must satisfy the following four properties:
||A|| > 0, unless A = 0 (in which case ||A|| = 0);

|| s * A || = |s| * ||A|| for any real number s;
|| A + B || <= ||A|| + ||B|| (triangle inequality);
|| A * B || <= ||A|| * ||B|| (submultiplicativity).
Matrix norms are most often needed when dealing with combinations of matrices and vectors. In such a
case, it is important that the matrix norm and vector norm that are being used are compatible.
Any given vector norm can be used to derive a corresponding matrix norm, guaranteed to be compatible.
This matrix norm is known as the vector-bound matrix norm.
Only if the matrix norm and vector norm are compatible can we write a useful bound like:
||A*x|| <= ||A|| * ||x||
Matrix norms include:
the L1 matrix norm;

the L2 matrix norm;
the L Infinity matrix norm;
the Frobenius matrix norm;
the spectral radius;
the EISPACK matrix norm;
Matrix Order
The order of a square matrix is the number of rows and columns. Thus a matrix of order 5 is a square
matrix with 5 rows and 5 columns.
The order of a rectangular matrix is described by giving both the number of rows and the number of
columns. Thus a rectangular matrix might have order 5 by 4.
The order of a matrix refers to a property of the mathematical object, which does not depend on how the
information is stored in a computer. The actual numbers representing the matrix may be stored in a
rectangular two dimensional array, whose row and column lengths are equal to or greater than the
mathematical orders, a rectangular array of lesser size (for band matrix storage, say), or even in a
collection of several separate one dimensional arrays.
Matrix Properties
Matrix properties are any features of a matrix which may be of use in choosing a storage scheme or
algorithm, or analyzing convergence or error properties, but which are not immediately evident from the
simple arrangement of zeroes in the matrix, or any symmetries among the nonzero elements.
Matrix properties include:
diagonal dominance;
invertibility;
irreducibility;
M matrix;
normality;
orthogonality;
positivity;
positive definiteness;
property A;
rank;
stochasticness;
Matrix Rank
The rank of a matrix is a measure of the linear independence of its rows and columns.
The row rank of a matrix of order M by N is the number of linearly independent rows in the matrix,
while the column rank is the number of linearly independent columns. The row rank will be between 0
and M, the column rank between 0 and N. If the row rank is equal to M, the matrix is said to have
maximal or full row rank; there are corresponding terms for column rank.
For a "square" matrix, of order N, the row and column ranks will be equal. A square matrix is
nonsingular if and only if it has maximal row rank; in other words, if no row is a linear combination of
the other rows, and similarly for columns. A square matrix with full rank has an inverse, a nonzero
determinant, and Gauss elimination with pivoting can be used to solve linear systems involving the
matrix.
Every singular matrix is "almost" nonsingular. That is, using any matrix norm you like, and no matter
how small you specify the (positive) tolerance epsilon, there is a nonsingular matrix closer than epsilon
to the singular matrix. Another way to look at this is to realize that it is always possible, by making tiny
changes to the entries of a singular matrix, to turn it into a nonsingular matrix. (Consider the zero
matrix; add epsilons to the diagonal entries, and it's nonsingular.) Thus, given the roundoff implicit in
computation, the determination of matrix rank is not a reliable process.
If the rank of a matrix is actually desired, a reasonable method is to compute the QR factorization. Very
small diagonal terms in the R factor may indicate linear dependence of the corresponding columns of the
matrix. The singular value decomposition will also give this sort of information.
Matrix Splitting
A matrix splitting is a decomposition of the system matrix:
A=M-N
in order to analyze the behavior of an iterative method for solving the linear system A*x = b. The M
matrix is the multiplier of the next iterate, and the N matrix is the multiplier of the current iterate.
For instance, consider the Jacobi iteration. If we decompose the matrix into its strictly lower triangular,
diagonal, and strictly upper triangular parts, we can write:

A=L+D+U
The Jacobi iteration can be written as:
D * xnew = - ( L + U ) * x - b
Hence, in the language of matrix splittings,
M=D
N=(L+U)
.
The matrix splitting gives us a convenient way of expressing the iteration matrix, which is the multiplier
of the current iterate in the (explicit) formula for the next one. In terms of the matrix splitting, this
iteration matrix always has the form M-1 * N. For instance, for the Jacobi iteration, the explicit formula
for xnew is:
xnew = - D-1 * ( L + U ) * x - D-1 * b
and so the iteration matrix is D-1 * ( L + U ).
Matrix Square Root

The square root of a matrix A is a matrix X with the property that
A = X * X.
For a matrix to have a square root, it must be symmetric. If a matrix is positive semidefinite, it is
guaranteed to have a square root. The square root can be determined from the Cholesky factorization as
follows:
A = L * LT
Now determine the Singular Value Decomposition of L
L = U * D * VT
and write
A = L * LT
= U * D * VT * ( U * D * VT )T
= U * D * VT * V * D * UT
= U * D * D * UT
= U * D * UT * U * D * UT
=X*X
where X is the desired matrix square root of A. In particular:
X = U * D * UT
Similarly, if B3=A, we say that B is the cube root or third root of A, and for any integer N, BN=A means
that B is called an N-th root of A.
Matrix Storage
Matrix storage formats are the schemes for storing the values of a mathematical matrix into memory
locations in a computer.
A special storage format may be chosen because of the overall matrix structure of zero entries, or
because of the matrix symmetry involving nonzero entries.
Common storage formats include:
Band Matrix Storage


Matrix Structure
Matrix structure is the classification of matrices according to the pattern of its zero entries.
If a matrix has a known structure, it may be possible to use a specialized storage scheme that takes less
space, or an algorithm that can execute more quickly.
A matrix about which nothing is known, or which exhibits no special pattern, may be called full or dense
or general.
Matrix structure patterns that can occur include:
band;
bidiagonal;
block;
diagonal;
Hessenberg;
rectangular;
sparse;
square;
trapezoidal;
triangular;
tridiagonal.
Matrix Symmetry
Matrix symmetry classifies certain common patterns that may relate the nonzero values of the matrix.
This classification occurs after the study of the basic matrix structure induced by the pattern of zero
elements.
As with matrix structure, any matrix symmetries that are present may influence the choice of storage
used for the matrix data, and the algorithms suitable to be applied to the matrix.
Matrix symmetries include:
anticirculant matrices;
antisymmetric matrices;
circulant matrices;
Hankel matrices.
Hermitian matrices;
persymmetric matrices.
skew Hermitian matrices;
symmetric matrices.
Toeplitz matrices.
Minimal Polynomial
The minimal polynomial of a matrix A is the monic polynomial P(X) of least degree with the property
that P(A)=0.
The Cayley-Hamilton theorem asserts that every matrix satisfies its own characteristic equation. In other
words, the polynomial
P ( lambda ) = det ( A - lambda * I ),
which is zero when lambda is equal to any of the numbers which are eigenvalues of the matrix A, is
equal to the zero MATRIX when lambda is replaced in the explicit formula for P(lambda) by A.
Thus, every matrix A of order N is guaranteed to be the root of a polynomial P(X) of degree N.
Therefore the minimal polynomial of A is either the characteristic equation, or else some polynomial Q
(X) of degree less than N. As a simple example, the characteristic polynomial of the identity matrix I is
XN-1, but the minimal polynomial is X-1.
The minimal polynomial of a matrix is used to define a derogatory matrix, which is important when
considering the eigenvalues and eigenvectors of a matrix.
Minor Matrix
A minor matrix is derived from a matrix A by removing some rows and columns. Usually both A and
the minor matrix are square, and usually only one row and one column are removed from A at a time.
For instance, if A is
1 2 3
4 5 6
7 8 9
then the minor matrix derived by deleting row 2 and column 1 is

2 3
8 9
The principal minor matrices or principal minors of a square matrix of order N are a set of N matrices
of orders 1, 2, ..., N; the M-th matrix has upper left entry A1,1 and lower right entry AM,M.
Simple facts involving minor matrices:
the cofactor matrix of A is derived by replacing each element of A by determinants of minor

matrices;
the determinant of A can be computed by an expansion that involves determinants of minor
matrices;
If the determinants of all the leading principal minors of a matrix are positive, then the matrix is
positive definite.
Monic Polynomial
A monic polynomial is a polynomial whose leading coefficient is 1.
The leading coefficient, of course, is the coefficient of the highest power of X, (or whatever the
independent variable happens to be). Thus, the polynomial X2+3*X+17 is monic, but X2+3*X3 is not.
The main reason for the idea of a monic polynomial is to be able to specify a unique polynomial with a
given property. Thus, for instance, there are many polynomials of degree 2 which are zero at 1, 2, and 3,
but they are all multiples of each other. In order to make a specific choice, we may specify that we mean
the monic polynomial with these properties.
Multiplicity
The algebraic multiplicity of a root lambda of a polynomial equation p(x)=0 is the "number of times
lambda is a root".
More precisely, the algebraic multiplicity is the exponent ma of the factor (x-lambda) in the
factorization of p(x). A root is known as a simple root if it has algebraic multiplicity 1; otherwise it is a
repeated root.
Eigenvalues are defined as the roots of the characteristic equation of a matrix, and so an eigenvalue has
an algebraic multiplicity. The behavior of the eigenvalue problem depends in part on the multiplicity of
the eigenvalues. In particular:
Every (distinct) eigenvalue has a corresponding linearly independent eigenvector;

If all eigenvalues are simple (ma=1), then the matrix is guaranteed to have a complete set of n
linearly independent eigenvectors;
the number of linearly independent eigenvectors for a given eigenvalue, sybmolized as mg, is
known as the geometric multiplicity of the eigenvalue, and represents the dimension of the linear
subspace spanned by those eigenvalues;
If an eigenvalue has algebraic multiplicity ma, it must be the case that 1 <= mg <= ma;
If, for at least one eigenvalue, mg < ma, then the matrix does not have a complete set of
eigenvectors, and is termed defective.
For example, for the matrix

7 1
0 7
7 is an eigenvalue of algebraic muliplicity 2, but there is only a single eigenvector x = ( 0, 1 ), so 7 has a

geometric multiplicity of 1.
Nilpotent Matrix
A nilpotent matrix is one for which the square, cube, or some finite power equals zero. For instance, any
strictly lower triangular matrix is nilpotent.
Consider the following matrix A and its powers:
0 0 0
= 2 0 0
3 4 0
0 0 0
A**2 = 0 0 0
8 0 0
0 0 0
A**3 = 0 0 0
0 0 0
Simple facts about a nilpotent matrix A:
the lowest power of A which equals 0 must be N or less, where N is the order of the matrix;
A is eigenvalue of A is zero.
Nonnegative Matrix
A nonnegative matrix A has only nonnegative entries; that is, for all indices I and J,
A(I,J) >= 0.
Similar terms, with obvious definitions, include matrices that are positive, negative and nonpositive. It's
easy to check if a matrix is nonnegative; this is much simpler than checking whether a matrix is positive
definite. The expression A >= 0 is sometimes used to express the notion that the matrix A is
nonnegative.
Facts about a nonnegative matrix A:
if x is nonnegative, then the vector A * x is nonnegative;

every nonnegative power AI is a nonnegative matrix.
if A is tridiagonal, all the eigenvalues are real;
A has at least one real, positive, eigenvalue, and a corresponding eigenvector with nonnegative
entries;
if A is also irreducible, then much more is known about its eigensystem. See the Perron
Frobenius Theorem.
Normal Equations
The normal equations for an M by N rectangular linear system
A*x=b
are computed by multiplying both sides by the transpose matrix:
AT * A * x = AT * b.
In the case where N < M, and the N columns of A are linearly independent, the matrix
B = AT * A
will be an invertible symmetric N by N matrix, and the normal equations can be solved by Gauss
elimination. The matrix B is sometimes called the Gram matrix or Gramian.
However, this method of producing an approximate solution of an overdetermined (and usually
inconsistent) linear system is usually not recommended. The matrix B has a condition number that is the
square of the condition number of A. Hence we may expect our solution to have a considerable error
component. We can do better. There are more accurate methods, such as the QR method, for which
more is known about the error growth, and the properties of the approximate solution that is produced.
Normal Matrix
A normal matrix A is a matrix that antisymmetric;
circulant;
diagonal;
Hermitian;
orthogonal;
Skew Hermitian;
symmetric;
unitary;
Note that an upper or lower triangular matrix is not normal, (unless it is actually diagonal!); such a
matrix may have a complete set of eigenvectors (for example, if the eigenvalues are distinct), but the
eigenvectors cannot be orthonormal.
Null Space
The null space of a matrix A is the set of all null vectors x such that
A * x = 0.
Simple facts:
The null space is a linear space;

A square matrix is nonsingular if and only if its null space is exactly the zero vector;
If x is a solution of A * x = b, then so is ( x + y ), where y is any vector in the null space of A;
linear solutions are only guaranteed to exist and to be unique when the null space is 0;
The eigenspace corresponding to an eigenvalue lambda is the null space of ( A - lambda * I ).
Null Vector
A null vector of a matrix A is a non-zero vector x with the property that A * x = 0.
Facts about a null vector:
A nonzero multiple of a null vector is also a null vector;

The sum of two null vectors is a null vector;
The set of all null vectors of a matrix forms the null space of the matrix;
If x solves A * x = b, and y is a null vector of A, then x + y is another solution, that is, A * ( x +
y ) = b;
A square matrix A is singular if and only if it has a null vector;
For a square matrix A, a null vector is an eigenvector for the eigenvalue 0.
Orthogonal Matrix
An orthogonal matrix A is a square, invertible matrix for which it is true that:
AT = A-1
which implies:
AT * A = A * AT = I.
Facts about an orthogonal matrix A:
Every row and column of A has unit Euclidean norm;

Distinct columns (or rows) of A are orthogonal;
The action of A on a vector is like a rotation or reflection;
For any vector x, ||A*x||2=||x||2;
The determinant of A is +1 or -1;
All eigenvalues of A have unit magnitude;
The singular values of A are all equal to 1;
The eigenvectors of A have unit length in the Euclidean norm;
The product of two orthogonal matrices is orthogonal;
The QR factorization of a rectangular M by N matrix A, with M>N has two forms:
Q is M by N, R is N by N;
Q is M by M, R is M by N;
For both factorizations, it is common to refer to the matrix Q as "orthogonal", but it is important to
realize that only a square matrix can be orthogonal. In the first factorization, the rectangular matrix Q
has columns that are of unit length pairwise orthogonal, and it is true that QT*Q=I, but the rows are
generally not of unit length, nor pairwise orthogonal, nor is it true that Q*QT=I.
In complex arithmetic, the corresponding concept is a unitary matrix.
Orthogonal Similarity Transformation

An orthogonal similarity transformation is a similarity relationship between two real matrices A and B,
carried out by an orthogonal matrix U, of the form:
A = U-1 * B * U = UT * B * U.
A and B are said to be orthogonally similar.
Orthogonal transformations are very common, particularly in eigenvalue computations. A general matrix
A may be reduced to upper Hessenberg form by such transformations, so that Q*A=B. The nice thing
about this is that if we later wish to reverse this transformation, the inverse of Q, because it is just the
transpose of Q! This means that orthogonal transformations are very easy to apply and invert.
Another nice feature of orthogonal transformations is that they may be built up gradually as the product
of a series of Householder matrices or Givens rotation matrices.
Orthonormal Vectors
A set of orthonormal vectors is a collection of M vectors X(I), each of order N, each of length 1 in the
Euclidean norm:
( X(I), X(I) ) = 1,
and pairwise orthogonal, so that if I and J are distinct:
( X(I), X(J) ) = 0.
Given any set of vectors, Gram Schmidt othogonalization or the QR factorization can produce an
orthonormal set of vectors, possibly fewer in number, that form a basis for the linear space that is
spanned by the original set.
Outer Product
An outer product of two vectors x of dimension M and y of dimension N is a matrix A of order M by N
whose entries are defined by:
A = x * yT,
or, entrywise:
A(I,J) = x(I) * y(J)
A matrix defined as the outer product of two vectors will usually have rank equal to 1. The only other
possibility is that the matrix might actually have rank 0. If an outer product is added to a matrix, this
operation is known as a rank one update. The Sherman Morrison Formula shows how to cheaply
compute the inverse of a matrix to which a rank one update has been applied.
Orthogonal Projector Matrix

An orthogonal projector matrix is idempotent and symmetric. Thus:
A*A=A
AT = A
Despite the name, an orthogonal projector matrix is in general not an orthogonal matrix. The only
exception is the identity matrix.
An overdetermined linear system A * x = b is, loosely speaking, a set of M linear equations in N
variables, with M > N.
Actually, it is possible for many of the equations to be redundant, so that what looks formally like an
overdetermined system can actually be determined or underdetermined.
Hence, a better definition is that an overdetermined linear system is a set of M inconsistent linear
equations in N variables, with M > N.
With this definition, a linear system is overdetermined exactly if it has no solution. In such a case, we
might be interested in a partial solution which satisfies as many equations exactly as possible, which we
might find simply by using pivoting to choose the equations to be satisfied, or a solution x which
minimizes the residual norm A*x-b, which we might find using the QR factorization.
Permanent of a Matrix
The permanent of a square matrix is a scalar value defined in a way similar to the determinant.
An explicit formula for the permanent of a matrix A is:
permanent ( A ) = sum [ over all P ] A(1,P(1)) * A(2,P(2) * ... * A(N,P(N)).
where the sum ranges over all possible permutations P of the numbers 1 through N. This differs from the
definition of the determinant only in that the sign of the permutation is not taken into account.
Permutation Matrix
A permutation matrix is a square matrix for which all the entries are 0 or 1, with the value 1 occuring
exactly once in each row and column. Such a matrix, when premultiplying or postmultiplying another
matrix, will simply permute the rows or columns of that matrix.
For example, the following is a permutation matrix:
0 1 0
0 0 1
1 0 0
If A is the matrix
11 12 13
21 22 23
31 32 33
then P*A permutes the rows of A:

21 22 23
32 32 33
11 12 13
while A*P permutes the columns:

13 11 12
23 21 22
33 31 32
Simple facts about a permutation matrix P:
P has a determinant of +1 or -1.

PT = P-1.
A permutation matrix that interchanges just two indices is an elementary matrix.
The use of pivoting during Gauss elimination means that along with the LU factors of A there is also a
permutation matrix factor P:
P * L * U = A.
Similarly, column pivoting may be used during the QR factorization of a matrix, and in that case, the
actual factorization is not A=Q*R, but A=Q*R*P, for some permutation matrix P.
Perron-Frobenius Theorem
The Perron-Frobenius Theorem tells us a great deal about the largest eigenvalue of a nonnegative
matrix.
The Perron-Frobenius Theorem:
If the nontrivial matrix A is nonnegative, then it has a (nontrivial) eigenvector x with
nonnegative entries, corresponding to a strictly positive eigenvalue lambda.
The Perron-Frobenius theorem is often applied to an irreducible matrix A, in which case the following
stronger statements apply:
the eigenvalue lambda is the largest eigenvalue of A in absolute value;

the eigenvalue lambda is simple (has algebraic multiplicity of 1);
the eigenvector x has no zero entries;
the eigenvector x is essentially unique (there are no eigenvectors of lambda that are linearly
independent of x).
When we say that A is nontrivial, we mean that there is at least one nonzero entry.
Persymmetric Matrix
A persymmetric matrix A is a square matrix whose values are "reflected" across its main anti-diagonal,
that is, for all I and J:

A(I,J) = A(N+1-J,N+1-I).
Here is an example of a persymmetric matrix:
4
7
9
10
3
6
8
9
2
5
6
7
1
2
3
4
Pivoting
Pivoting is the attempt to improve accuracy by choosing the most suitable column or row for use during
a single step of a linear algebra algorithm.
The most common instance of pivoting occurs in Gauss elimination. During the first step, we wish to
eliminate all the entries in column one except for one entry. We will do this by adding carefully chosen
multiples of row one to the other rows. For example, if the first two rows were
2 3 5 8
6 8 9 2
we could add -3 times row 1 to row 2, and thus zero out the "6" in the first column of row 2. This
scheme would not work, however, if the first entry in the first row were zero, because no multiple of
zero can be anything but zero.
Thus, we allow ourselves to interchange the first row with some other row which has a nonzero entry in
the first column. But since we're going to interchange rows anyway, it turns out that best accuracy
occurs if we choose to bring in the entry with the largest absolute value. Such a scheme, which considers
each column in order, and uses the largest entry in the row as the pivot, is called partial pivoting. This
form of pivoting is used in most linear algebra software.
It can be shown that Gauss elimination can be carried out without pivoting if each of the principal
minors of the matrix is nonsingular. If the matrix is positive definite, then this condition is guaranteed,
and it is common to factor the matrix without pivoting.
A complete pivoting or full pivoting scheme would search for the entry of largest magnitude anywhere in
the uneliminated portion of the matrix, and use that entry as the next pivot. Such a scheme requires a lot
more searching, and auxiliary storage of the row and column numbers of the pivots. It is little used, since
it does not seem to bring a great improvement in accuracy.
QR factorization can also use pivoting. At step I, columns I through N are examined. The column with
the greatest norm is interchanged with column I, and then the QR operations for that step are carried out.
In this case as well, pivoting is done to try to ensure stability.
Polar Decomposition
The polar decomposition of any matrix A has the form
A=P*Q
where P is positive semidefinite and has the same rank as A, and Q is unitary.
The polar decomposition of a matrix can be determined from its singular value decomposition:
A = U * D * VT
= ( U * VT ) * ( V * D * VT )
=P*Q
where we have
P = U * VT
Q = V * D * VT
Positive definite matrix

A (complex) matrix A is positive definite if it is true that for every nonzero (complex) vector x, the
product:
x* * A * x > 0,
where, in particular, we are requiring that this product be a real number.

A real matrix A is restricted positive definite if it is true that for every nonzero real vector x:
xT * A * x > 0.
If a real matrix is positive definite, then it is restricted positive definite.
Every complex positive definite matrix is guaranteed to be Hermitian. Thus, the phrase positive definite
Hermitian is, strictly speaking, redundant. However, a restricted positive definite matrix is not
guaranteed to be symmetric! This can happen because we only use real vectors in the test product.
Here is a simple example of a restricted positive definite matrix which is not symmetric:
1
-1
1
1
because xT * A * x = x12+x22
Whenever real positive definite matrices occur in practical applications, however, they are assumed or
required to be symmetric. If necessary, one can decompose a positive definite matrix in the restricted
sense into its symmetric and antisymmetric parts. The symmetric part will actually be (fully) positive
definite.
Simple facts about a positive definite matrix A:
A is nonsingular;
The inverse of A is positive definite;
every eigenvalue of A is positive;
Gauss elimination can be performed on A without pivoting;
if A is symmetric as well, it has a Cholesky factorization A = L * LT, where L is lower
triangular.
A matrix about which no such information is known is called indefinite. A matrix for which the product
is always nonnegative is sometimes called positive indefinite or positive semidefinite. There are similar
definitions of negative definite and negative indefinite.
Positive Matrix
A positive matrix A has only positive entries; that is, for all indices I and J,
A(I,J) > 0.
Similar terms, with obvious definitions, include matrices that are nonnegative, negative and nonpositive.
It's easy to check if a matrix is positive; this is much simpler than checking whether a matrix is positive
definite.
Simple facts about a positive matrix A:
A is a nonnegative matrix;
the spectral radius of A is positive (in other words, A is not nilpotent!);
every positive power AI is a positive matrix.
The Power Method

The power method is a simple iterative algorithm for computing the eigenvalue of largest magnitude in a
matrix, and its corresponding eigenvector.
The algorithm for an arbitrary matrix A works as follows:
Initialize the vector x arbitrarily;

Divide each entry of x by ||x||.
Compute xnew = A * x.
Compute lambda = ||xnew|| / ||x||.
If the estimated eigenvalue has changed significantly since the last iteration, then replace x by
xnew, and repeat the preceding steps.
The speed of convergence of the algorithm depends largely on the ratio between the magnitude of the
largest and the second largest eigenvalues. If the second largest eigenvalue is close in magnitude to the
largest, convergence will be slow. If A has several distinct eigenvalues of the same magnitude, say -2
and 2, or 5, 3+4i and 3-4i, then the algorithm will fail.
While this algorithm is very easy to program, it is difficult to adapt it to the case where other
eigenvalues are desired. The use of deflation is a possibility, but still forces the user to compute the
eigenvalues one at a time, and in order of size, making it rather difficult to find, say, just the smallest
eigenvalue.
The inverse power method, a generalization of the power method, has the advantage of being able to
find the other eigenvalues of the matrix.
Preconditioner
A preconditioner is a matrix M used to improve the performance of an iterative method for linear
equations.
The simplest preconditioning is done using a left preconditioner. The original linear system is leftmultiplied by the inverse of the preconditioning matrix, resulting in the system
M-1*A*x=M-1*b
For more complicated preconditioning, both left and right preconditioners, M1 and M2 may be used, so
that the system is transformed to
M1-1*A*M2-1*M2*x=M-1*b
A left and right preconditioning like this can be used if the original matrix is symmetric, and it is desired
to preserve this symmetry in the transformed system.
The convergence of the iterative scheme depends on properties of the system matrix. A suitable choice
of a preconditioner can mean that the transformed problem converges much more rapidly than the
original one.
Although the definition of the preconditioner matrix suggests that we have to compute its inverse, it is
usually the case that we have a factorization of the preconditioner which allows us to solve linear
systems. Any computations that formally involve the inverse can therefore be replaced by linear system
solves.
Desirable properties of a preconditioner matrix include:
M should approximate A in some way;

It should be significantly easier to solve linear systems involving M than A (otherwise the
problem has just gotten harder);
M should not require much more storage space than A.
Examples of preconditioners include
the Jacobi preconditioner;

the incomplete LU factorization;
the incomplete Cholesky factorization;
Property A
A matrix A has property A if the indices 1 to N can be divided into two sets S and T so that, for any A(I,
J) which is not zero, it must be the case that:
I = J, or
I is in S and J is in T, or
I is in T and J is in S.
This is equivalent to saying that, by listing the S rows and columns first, the matrix can be rewritten in
the block form:
| D1
| G
F |
D2 |
where D1 and D2 are diagonal.

The Pseudoinverse
The pseudoinverse is a generalization of the idea of the inverse matrix, for cases where the standard
inverse matrix cannot be applied. Such cases include matrices A which are singular, or rectangular.
The pseudoinverse is sometimes called the Moore Penrose inverse or the generalized inverse.
The pseudoinverse of an M by N rectangular matrix A is defined as the unique matrix N by M matrix A
+ which satisfies the four conditions:
A * A+ * A = A
A+ * A * A+ = A+
(A * A+)T = A * A+
(A+ * A)T = A+ * A
Note that if A is a square invertible matrix, then the pseudo inverse is actually the inverse.
The pseudoinverse can be used in a way similar to the way an inverse is used. For instance, given the
rectangular set of linear equations
A*x=y
a "solution" can be computed as:
x = A+ * y.
If the equations are consistent, then x will actually satisfy the equations. Otherwise, x will be a "best
possible" solution, in the sense that it minimizes the Euclidean norm of the residual error.
The pseudoinverse can be computed from the information contained in the singular value
decomposition, which has the form:
A = U * S * VT
where
A is an M by N rectangular matrix,
U is an M by M orthogonal matrix,
S is an M by N diagonal matrix,
V is an N by N orthogonal matrix.
The formula for the pseudoinverse of A is then:

A+ = V * S^ * UT
where S^ is an M by N diagonal matrix, whose diagonal entries are the inverse of the diagonal entries of
S, except that where S has zero entries, so does S^.
QR Factorization
The QR factorization factors a matrix A into an orthogonal matrix Q and an upper triangular matrix R,
so that
A = Q * R.
The factorization can also be applied to rectangular matrices, in which case one of the factors is no
longer square.
The QR factorization can be useful for solving the full variety of linear systems, whether nonsingular,
under-determined, over-determined or ill conditioned. It can be used to carry out the Gram Schmidt
orthogonalization of a set of vectors constituting the columns of A. The QR factorization is also used
repeatedly in an iterative solution of eigenvalue problems.
The QR factorization can be produced incrementally, by a series of transformations involving
Householder matrices or Givens rotation matrices
As an example of QR factorization, the matrix A:
1 1 0
1 0 1
0 1 1
can be factored into the orthogonal matrix Q:

SQRT(1/2) SQRT(1/6) -SQRT(1/3)
SQRT(1/2) -SQRT(1/6) SQRT(1/3)
0
SQRT(2/3) SQRT(1/3)
and the upper triangular matrix R:

SQRT(2) SQRT(1/2) SQRT(1/2)
0
SQRT(3/2) SQRT(1/6)
0
0
SQRT(4/3)
LAPACK and LINPACK include routines for computing the QR factorization.

QR Method for Eigenvalues

The QR method is used to find the eigenvalues of a square matrix.
For efficiency, the method begins by transforming the matrix A using Householder matrices until we
have determined an upper Hessenberg matrix A1 which is orthogonally similar to A.
The QR factorization of A1 is then computed, A1 = Q1*R1. Then the factors Q1 and R1 are reversed, to
compute a new matrix A2 = R1*Q1. A2 itself can be QR factored into A2 = Q2*R2, and then reversal
of factors results in a matrix A3, and so on.
Each matrix A1, A2, and so on is orthogonally similar to A, and so shares the same eigenvalues. The
sequence of matrices A, A1, A2, A3, ... will generally converge to a matrix B of a very simple type: B
will consist entirely of diagonal entries and two by two blocks. The diagonal entries are the real
eigenvalues, and the eigenvalues of the 2 by 2 blocks are the complex eigenvalues.
In the particular case where the eigenvalues are all real, (and hence B is diagonal), and if the orthogonal
transformations have been accumulated along the way, the result can be written:
A = Q * B * QT
which is an orthogonal similarity transformation of A into a diagonal matrix, or
A*Q=Q*B
which means that the columns of Q are the eigenvectors.
If the matrix B is not diagonal, then the pair of columns corresponding to any 2 by 2 block can be used
to construct the pair of corresponding complex eigenvectors. Alternatively, once the eigenvalues have
been determined, the inverse power method can be used to compute any particular eigenvector.
Quaternion Representation
Quaternions, discovered by William Hamilton, have the form a+bi+cj+dk, where i, j and k are "special"
quantities.
The properties of "1" and the 3 special quantities are best displayed in a multiplication table:
1 i j k
1 1 i j k
i i -1 k -j
j j -k -1 i
k k j -i -1
It is possible to devise matrices that behave like quaternions. Let the value "1" be represented by the
identity matrix of order 4, and the value "i" be represented by
0
-1
0
0
1
0
0
0
0 0
0 0
0 -1
1 0
the value "j" by

0 0
0 0
-1 0
0 -1
1
0
0
0
0
1
0
0
the value "k" by

0
0
0
-1
0 0
0 -1
1 0
0 0
1
0
0
0
Then it is easy to show that these matrices, and linear combinations of them, obey the rules of
quaternions.
Rayleigh Quotient
The Rayleigh quotient of a matrix A and vector x is
R(A,x) = (xT*A*x) / (xT*x).

If the matrix A is symmetric, then the Rayleigh quotient is an excellent estimate of the "nearest"
eigenvalue; in particular,
Lambdamin <= R(A,x) <= Lambdamax
On the other hand, if the matrix is not symmetric, then we cannot guarantee that the eigenvalues will be
real; moreover, we cannot guarantee that the eigenvectors can be assumed to be orthogonal. Thus for a
general matrix, the information obtained from the Rayleigh quotient must be interpreted more carefully.
A frequent use of the Rayleigh quotient is in the inverse power method. Starting with approximations for
the eigenvalue and eigenvector, one step of the inverse power method gives an improved estimate of the
eigenvector. Then the Rayleigh quotient improves the eigenvalue estimate, and will be used to shift the
next step of the inverse power method. This pair of steps can be rapidly convergent.
If the matrix A is actually positive definite symmetric, then the quantity (yT*A*x) has all the properties
of an inner product. In that case, the Rayleigh quotient may be generalized to involve any pair of
vectors:
R2(y,A,x) = ( yT*A * x ) / ( yT*x ).
If the matrix is not symmetric, but we still want a sensible way to estimate the eigenvalues, then the
"Unsymmetric Rayleigh Quotient" could be defined as:
URQ(A,x) = sqrt ( ( ( A * x)T*(A * x) ) / ( xT*x ) ).
This amounts to the square root of the Rayleigh quotient for the symmetric matrix AT*A, but can also be
regarded as the ratio of the L2 norms of A*x and x.
Rectangular Matrix
A rectangular matrix is a matrix which is not "square", that is, a matrix whose row order and column
order are different.
While many operations and algorithms of linear algebra only apply to a square matrix, a rectangular
matrix does have an LU factorization, and a singular value decomposition.
A rectangular matrix can occur when solving an under-determined or over-determined linear system.
Reflection Matrix
A reflection matrix A has the property it carries out the reflection (or negation) of the portion of every
vector that is perpendicular to some hyperplane, while leaving the parallel portion of the vector
unchanged.
A reflection matrix A is involutory, that is,
A * A = I,
which strongly restricts the eigenvalues.
Examples of reflection matrices include the identity matrix, a diagonal matrix whose diagonal entries are
+1 or -1, any matrix which rotates two coordinate axes by 180 degrees, and the Householder matrices.
Simple facts about a reflection matrix A:
A = A-1;
the matrix I - A is an idempotent matrix;
All eigenvalues of A have magnitude 1.
Residual Error
The residual error is a measure of the error that occurs when a given approximate solution vector x is
substituted into the equation of the problem being solved.
For a system of linear equations, the residual error is the vector r defined as
r=b-A*x
For the eigenvalue problem, the residual error is a vector which is a function of the approximate
eigenvector x and the approximate eigenvalue lambda:
r = lambda * x - A * x.
When carrying out an iterative solution process, it is common to compute the residual error for each new
approximate solution, and to terminate the iteration successfully if the vector norm of the residual error
decreases below some tolerance.
An important fact to realize is that the residual error is not, by itself, a reliable estimate of the error in the
solution of a linear system. If the residual error has small norm, we can really only hope that the solution
error (between our computed x and the true solution x*) is small. We need to know the norm of the
inverse of the matrix, in which case the following restriction holds:
||x - x*|| <= ||A-1|| * || A ( x - x* ) ||
= ||A-1|| * || A * x - b + b - A * x* ||
= ||A-1|| * ||r||,
so that when the norm of the residual error, ||r|| is small, we have a precise upper bound on the error in
the solution. Since such matrix norms are generally not known or computable, what we really have is a
promise of continuity in the errors: as we drive the residual down, we are forcing down the upper limit
on the approximation error.
Root of Unity
An N-th root of unity is any complex number W such that WN = 1.
For a given N, there are N such roots, which can be summarized as:
THETA = 2 * pi / N
WJ = cos ( ( J - 1 ) * THETA ) + i * sin ( ( J - 1 ) * THETA )
Roots of unity are especially useful in discussing Fourier matrices and Circulant matrices.
Rotation
A rotation is a linear transformation which preserves (Euclidean) distances.
Because a rotation R is a linear transformation, the value of R * 0 must be 0; in other words, the origin
does not move. Because a rotation preserves distances, it must be the case that, in the Euclidean vector
norm,
|| R * x || = || x ||
for every vector x in the space. From this fact, we can conclude that:
R has an L2 matrix norm of 1;

every eigenvalue of R has magnitude 1.
Common examples of matrices which embody rotations include:
the identity matrix;

a Givens rotation matrix;
any orthogonal matrix;
any Hermitian matrix.
Row Echelon Form

Row echelon form is a special matrix structure which is usually arrived at by a form of Gauss
elimination.
Any matrix, including singular and rectangular matrices, can be transformed into this form, using a
series of elementary row operations. Once the form is computed, it is easy to compute the determinant,
inverse, the solution of linear systems (even for underdetermined or overdetermined systems), the rank,
and solutions to linear programming problems.
Moreover, the process can be considered a matrix factorization, of the form
A=B*E
where B is nonsingular, and E is in row echelon form.
A matrix (whether square or rectangular) is in row echelon form if:
Each nonzero row of the matrix has a 1 as its first nonzero entry.
The leading 1 in a given row occurs in a column to the right of the leading 1 in the previous row.
Rows that are completely zero occur last.
A matrix is in row reduced echelon form if it is in row echelon form, and it is also true that:
Each column containing a leading 1 has no other nonzero entries.
Row echelon form is primarily of use for teaching, and analysis of small problems, using exact
arithmetic. It is of little interest numerically, because very slight errors in numeric representation or
arithmetic can result in completely erroneous results.
Row Rank
The row rank of a matrix is the number of linearly independent rows it contains.
The matrix
1 2 3 4
5 6 7 8
9 10 11 12
has row rank 2, because row 3 is equal to twice row 2 minus row 1.
For any matrix, the row rank is the same as the column rank. This common value is also equal to the
rank of the matrix, which is defined for both square and rectangular matrices.
For a square matrix of order n, the rank is a number between 0 and n. A square matrix whose rank is n is
said to be nonsingular. For a rectangular matrix of order m by n, the rank must be a number between 0
and the minimum of m and n. Rectangular matrices which attain this maximal value are not called
"nonsingular". Instead, they are said to have full row rank, full column rank, or maximal rank.
Row Space
The row space of an M by N matrix A is the set of all possible linear combinations of rows of A.
If the N vector v is a linear combination of rows of A, then there is a M vector c of coefficients with the
property that
v=c*A
In other words, the row space is the set of all possible results of premultiplying A by an arbitrary vector.

The Schur decomposition of a complex square matrix A is a factorization of the form
A = U * T * U*
where T is an upper triangular matrix, and U is unitary. The decomposition is also known as the Schur
normal form. The matrix T is similar to A.
If A is actually a real matrix, then it has the unitary decomposition just described, but it also has an
orthogonal decomposition that is "almost" upper triangular:
A = Q * T2 * QT
where T2 is a block upper triangular matrix, and Q is orthogonal. The diagonal blocks of T2 are either
singletons, corresponding to real eigenvalues, or 2 by 2 blocks, corresponding to complex conjugate
pairs of complex eigenvalues.
In particular, if A is real and has all real eigenvalues, then the matrix T2 must be diagonal, and so the
decomposition has the form:
A = Q * LAMBDA * QT
where LAMBDA is the diagonal matrix of eigenvalues, and Q is the eigenvector matrix.
Some facts about the Schur decomposition of a matrix A:
Every square matrix has a Schur decomposition;

The eigenvalues of A are the diagonal elements of T;
If A is Hermitian, then T is diagonal.
A is normal if and only if T is diagonal.
If T is diagonal, then the rows of U are the eigenvectors of A.
Sherman Morrison Formula

The Sherman Morrison formula applies to a matrix which has been perturbed in a simple way, and
produces the inverse, or the solution of a linear system for the perturbed system, based on information
already computed for the original system.
The perturbation of the coefficient matrix A is assume to be of the form:
C = A + x * yT,
where the perturbation is an outer product of the vectors x and y. Changing a single entry A(I,J) can be
represented in this way, for instance.
The Sherman Morrison formula for the inverse of the perturbed matrix is:
C-1 = A-1 - z * wT / ( 1 + alpha ),
where:
z = A-1 * u;
w = A-T * v;
alpha = vT * z.
In the common case where the solution of C * x = b is desired, and the LU factorization of A has been
computed, so that linear systems involving the original matrix A can be easily solved, the procedure is:
solve A * z = u for z;
solve AT * w = v for w;
set alpha = vT * z;
set beta = wT * b;
solve A * x = b for x;
adjust x := x - beta * z / ( 1 + alpha ).
The method will fail if 1 + alpha is 0, which can indicate singularity of the matrix C or a more technical
problem.
Sign Symmetric Matrix

A sign symmetric tridiagonal matrix A is one for which the signs of every pair of corresponding offdiagonal entries are equal:
sign ( A(I,I+1) ) = sign ( A(I+1,I) )
for I = 1 to N-1. (A value of zero matches any sign of the other entry.)
A symmetric tridiagonal matrix is always sign symmetric. Some EISPACK routines handle the case of a
sign symmetric tridiagonal matrix directly. A matrix is strictly sign symmetric if it is sign symmetric and
zero entries only occur in pairs.
The following tridiagonal matrix is not symmetric, is sign symmetric, and not strictly sign symmetric:
1 2 0
3 4 -7
0 -3 4
0 0 5
0 0 0
0
0
2
3
1
0
0
0
0
9
Similar Matrix
Two matrices A and B are similar if B is related to A by a matrix P in the following way:
B = P-1 * A * P.
In this case, P is said to be the similarity transformation matrix.
Matrices which are similar have the same eigenvalues. Special cases include the similarity matrix P
being an elementary transformation, or orthogonal or unitary.
Many algorithms try to improve speed or efficiency by using similarity transforms on an input matrix A,
so as to find a simpler matrix B for which the problem can be more easily or more quickly solved. It
may then be necessary to take the answer for the problem about B and "backtransform" it to an answer
for the problem about A. For example, if we get the eigenvalues and eigenvectors of B, A will have the
same eigenvalues, but will have different eigenvectors, related by the similarity transform.
Every symmetric matrix A is orthogonally similar to a diagonal matrix. Since the inverse of an
orthogonal matrix is its transpose, this relationship may be written
B = Q-1 * A * Q = QT * A * Q.
Singular Matrix
A singular matrix is a square matrix that does not have an inverse.
Facts about a singular matrix A:
A has a null vector.

A has a zero eigenvalue.
The determinant of A is zero.
There is at least one nonzero vector b for which the linear system A*x=b has no solution x.
The singular linear system A*x=0 has a nonzero solution x.

The singular value decomposition of a rectangular M by N matrix A is a factorization of the form:
A = U * S * VT
where, in the EISPACK version:
U is an M by N matrix with orthogonal columns;

S is an N by N diagonal matrix, containing the nonnegative singular values of A;
or, in the LINPACK version:
U is an M by M orthogonal matrix;
S is an M by N rectangular matrix whose diagonal contains the nonnegative singular values of A;

The LINPACK form has the advantage that U and V are both orthogonal, and S retains the "shape" of A.
Moreover, this format allows us to consider U and V to be composed of left and right singular vectors,
in analogy to the factorization of a square matrix via two orthogonal matrices of left and right
eigenvectors.
The solution of A * x = b for non-square A can be found by seeking that x which minimizes the ||A*xb||. That x is equal to
V * S-1 * UT,
where, since S may have zeroes on its diagonal, S-1 is constructed by replacing each nonzero diagonal
element by its inverse.
For any column I no greater than the minimum of M and N, let ui be the I-th column of U, and vi be the
I-th column of V, and sii be the I-th diagonal element of S. Then it is a fact that
A * vi = sii * ui
and
AT * ui = sii * vi
which allows us to conclude that
A * AT * ui = sii * sii * ui
and
AT * A * vi = sii * sii * vi
In other words, U, V and S contain information about the eigenvalues and eigenvectors of A * AT.
Conversely, if we know the eigenvalues and eigenvectors of A * AT, then we know the squares of the
singular values of A, and the left singular vectors of A (the U matrix).
The singular value decomposition can be used to construct the the pseudoinverse of the rectangular or
singular matrix A.
Routines for singular value decomposition are included in EISPACK, LAPACK, and LINPACK.
Skew CentroSymmetric Matrix

A skew centrosymmetric matrix is one which is antisymmetric about its center; that is,
Ai,j = - Am+1-i,n+1-j
Example:
1 10
-13
2
-6 -7
12 -4
5 -11
8 11 -5
9
4 -12
0
7
6
-9 -2 13
-8 -10 -1
Facts about a skew centrosymmetric matrix A:
If lambda is an eigenvalue of A, then so is -lambda;

If x is an eigenvector of A, then so is J*x where J is the Exchange matrix;
Skew Hermitian Matrix

A complex matrix A is skew Hermitian if it is equal to the negative of its transpose complex conjugate:
A = - ( conjugate ( A ) )T.
A skew Hermitian matrix must have a purely imaginary diagonal.

Skyline storage is a matrix storage method for storing a particular kind of sparse matrix. The format is
simple, compact, and suitable for use with Gaussian elimination.
Skyline storage is most typically used with symmetric matrices derived from a finite element problem.
In this setting, it is common to encounter a matrix which is "nearly" banded, or "raggedly banded". That
is, the nonzero elements of the matrix are always near the diagonal, but the number of such elements
varies from column to column.
Here is a matrix suitable for skyline storage:
A11 A12 0 A14 0
A21 A22 0 A24 0
0
0 A33 0
0
A41 A42 0 A44 A45
0
0
0 A54 A55
If the matrix is symmetric, we can regard the matrix as a sequence of columns, starting with the first
nonzero element in a column, and proceeding to the diagonal, including every entry in between, whether
or not it is zero. Thus, this matrix could be regarded as equivalent to the following five column vectors:
A11
A12
A22
A33
A14
A24
A34
A44
A45
A55
(Note that we have added a location for A34. It's storing a zero, but it's between a nonzero entry, A24,
and the diagonal A44). This is the heart of the idea behind skyline storage. We simply cram all these
columns into a single vector:
A11, A12, A22, A33, A14, A24, A34, A44, A45, A55,
and then figure out a way to address the individual entries. The obvious route is to have a way of
pointing to the diagonal entries:
A11
A22
A33
A44
A55
is
in
in
in
in
in entry 1,
entry 3,
entry 4,
entry 8,
entry 10.
and we can store these values in an array called "INDEX". Then, we know that column 4 of the matrix is
contained in locations INDEX(3)+1 through INDEX(4).
The reason that skyline storage is ideal for Gaussian elimination is that we have already set aside all the
entries we will ever need to take care of "fill in", as long as we are not performing any pivoting.
For instance, consider what happens when we are eliminating row 1. We will add multiples of row 1 to
rows 2 and 4. If there is a nonzero entry in a particular column of row 1, then that entry could cause "fill
in" when added to row 2 or 4. But there is no problem, because if column J of row 1 is nonzero, then
we're sure we've already set aside a entry for column J of rows 2 through row J. So we will never have to
modify our storage scheme because of unexpected nonzero entries generated by fillin.
On the other hand, it should be clear that indexing entries of the matrix can be tedious or cumbersome.
In particular, it is much harder to find entries of a row of the matrix than of a column. Writing a routine
to actually carry out Gauss elimination with such a data structure can be something of a trial.
Thus skyline storage achieves simplicity and compactness in storage, at the cost of complexity in
coding.
Span of a Set of Vectors

The span of a set of vectors { v(i)| 1 <= I <= N } is the set of vectors that can be constructed from the set
via linear combinations. The span is sometimes called the spanning space.
The span of a set of vectors is a linear space; it includes the 0 vector, each of the original vectors, and
sums and multiples of them.
As an example of a vector span, consider the eigenvectors associated with a particular eigenvalue of a
matrix. If the eigenvalue has algebraic multiplicity of 1, then we usually think there is only one
associated eigenvector; if the eigenvalue has a higher multiplicity, there may be 2 or more linearly
independent eigenvectors. In any case, the eigenspace associated with the eigenvalue is the span of these
eigenvectors. Any element of this span is itself also an eigenvector for the eigenvalue.
Given a set of vectors which define a spanning space, it is often desired to know whether a subset of
those vectors will define the same spanning space; that is, whether we can toss out some of the vectors
because they don't add anything to the span. If we toss out as many vectors as possible, we end up with a
basis for the span; the number of vectors remaining tells us the dimension of that space.
The Sparse BLAS

The Sparse BLAS are a set of vector oriented linear algebra routines useful for sparse matrix problems.
The success of the Basic Linear Algebra Subprograms (BLAS) motivated the creation of a small set of
similar routines for use with sparse vectors. The routines enable the interaction of sparse and full
vectors. The sparse vectors are assumed to be stored as a vector of values, and a vector of indices. For
example, a sparse vector might be represented by the pair
X = (1.0, 2.0, 3.0)
IX = (9, 1, 200)
which represents a vector that is entirely zero, except that entry 1 equals 2.0, entry 9 equals 1.0 and entry
200 equals 3.0.
FORTRAN subroutines which implement the sparse BLAS are available through the NETLIB web site.
The single precision Sparse BLAS routines include:
SAXPYI adds a multiple of a sparse vector to a dense vector.

SDOTI computes the dot product of a dense and a sparse vector.
SGTHR gathers entries of a dense vector into a sparse vector.
SGTHRZ gathers entries of a dense vector into a sparse vector, zeroing first.
SROTI applies a Givens rotation to a sparse vector.
SSCTR scatters entries of a sparse vector into a dense vector.
Sparse Matrix
A sparse matrix is a matrix with so many zero entries that it is profitable to use special schemes to store
the matrix and solve problems involving it. A rule of thumb is that if less than 5% of the matrix is
nonzero, the matrix can be called sparse.
Not only does a sparse matrix have few nonzeroes, but these nonzeroes are typically scattered seemingly
at random throughout the matrix. Thus, unlike a band matrix, (which also is "mostly zero") we have to
expend a significant amount of effort simply recording where every nonzero entry occurs.

A sparse storage scheme is a matrix storage method of storing the nonzero entries of a sparse matrix in a
convenient and efficient way.
There are many different kinds of sparse matrix storage scheme. Their features depend on the properties
of the underlying problem. In this discussion, a few simple non-specific schemes will be discussed.
If a direct linear equation solver will be used, storage must also be made available for "fill-in" values
that will occur during the factorization process. An iterative solver, on the other hand, has no fill-in
values, and never needs to change the entries of the matrix.
A simple storage scheme would require the user to supply NZ, the number of nonzero entries, and three
arrays of size NZ:
IROW(I) and ICOL(I) specify the row and column indices for entry I;
A(I) specifies the value of entry I. Thus the first nonzero entry of the matrix occurs in row IROW
(1), column ICOL(1), with the value A(1).
If this scheme seems excessive, the elements of the matrix could be listed in order of rows, the ROW
array could be replaced by a ROWEND array of length N. ROWEND(I) is the index in A of the last
nonzero element of row I of the original matrix. This small saving comes at the price of complicating
access to the matrix.
SPARSKIT
SPARSKIT is a tool kit for sparse matrix computations.
SPARSKIT can manipulate sparse matrices in a variety of formats, and can convert from one to another.
For example, a matrix can be converted from the generalized diagonal format used by ELLPACK and
ITPACK to the format used by the Harwell-Boeing Sparse Matrix Collection (HBSMC) or even into
LINPACK banded format.
Utilities available include converting data structures, printing simple statistics on a matrix, plotting a
matrix profile, performing basic linear algebra operations (similar to the BLAS for dense matrix), and so
on.
Spectral Radius
The spectral radius of a matrix is the magnitude of the largest eigenvalue of a matrix.
The spectral radius is often easy to compute, and it is a useful measure of the "size" or "strength" of a
matrix. However, the spectral radius is not a vector-bound matrix norm; it is not compatible with any
vector norm.
Very simply, we're saying that there is no vector norm for which it will be true, for all vectors x, that:
|| A * x || <= ||A|| * ||x||
if the matrix norm used is the spectral norm.
To see this, consider a matrix whose dominant eigenvalue lambda > 0 has algebraic multiplicity strictly
greater than its geometric multiplicity. Then there must be an eigenvector x so that A * x = lambda * x,
but there is also a generalized eigenvector y, orthogonal to x, with the property that A * y = x + lambda
* y. Now, if the spectral radius is a vector-bound matrix norm, then it must be the case that
|| A * y || <= | lambda | * || y ||,
but, since x is orthogonal to y, we can show that:
|| A * y || = || x + lambda * y || > | lambda | * || y ||.
Hence, the spectral radius is not a vector-bound matrix norm.
On the other hand, the value of the spectral radius is a lower bound for the value of any vector-bound
matrix norm on A, because there must be an eigenvalue lambda and a vector of unit norm x with the
property that
A * x = lambda * x
so the norm of A*x divided by the norm of x is lambda. Therefore, the matrix norm of A must be at
least | lambda |.
The Euclidean norm of a real symmetric matrix is equal to its spectral radius.
Spectrum
The spectrum of a square matrix is the set of eigenvalues.
Square Matrix
A square matrix is one which has the same number of rows and columns. Most cases (but not all!)
requiring the solution of a linear system involve a square coefficient matrix.
Only a square matrix has a determinant, an inverse (if not singular!) a trace, powers, and eigenvalues.
If a matrix is not square, it is called rectangular.
Stochastic Matrix
A stochastic matrix has only nonnegative entries, with the entries in each row summing to 1.
A stochastic matrix may also be called a row stochastic matrix or a transition matrix. These names
derive from the fact that the entry A(I,J) may be viewed as a probability that a system currently in state I
will transition to state J on the next step.
A column stochastic matrix has only nonnegative entries, with the entries in each column summing to 1.
A doubly stochastic matrix is both row and column stochastic.
Facts about a stochastic matrix A:
A is a nonnegative matrix;
Every eigenvalue of A must be no greater than 1 in modulus;
If A and B are stochastic, then so is A * B;
If A and B are doubly stochastic, then so is A * B;
If A is doubly stochastic, then A can be written as
A = Sum ( 1 <= I <= N ) c(I) * P(I)

where Nis the order of A, the real numbers c(I)sum to 1, and each P(I)is a permutation matrix.
An ergodic matrix is a row stochastic matrix which has no eigenvalues of modulus 1 except for 1 itself.
Strassen's Algorithm
Strassen's algorithm for matrix multiplication is a method which can produce a matrix product more
efficiently than the standard approach.
For simplicity, suppose that the two factors A and B are both of order N. Then the obvious method of
producing the product C requires N multiplications and N additions for each entry, for a total of 2 * N3
floating point operations.
Strassen's algorithm is defined recursively. It is easiest to describe if the matrix has an order that is a
power of two. In that case, the product of two matrices of order N is described in terms of the product of
matrices of order (N/2), and so on, until factors of order 2 are reached.
Now suppose that A and B are each of order 2. The definitions for the entries of the product C are:
C11 = A11 * B11 + A12 * B21
C21 = A21 * B11 + A22 * B12
C12 = A11 * B12 + A12 * B22

C22 = A21 * B12 + A22 * B22
Now compute the following quantities:

P1
P2
P3
P4
P5
P6
P7
=
=
=
=
=
=
=
(A11+A22) * (B11+B22)
(A21+A22) * B11
A11 * (B12-B22)
A22 * (B21-B11)
(A11+A12) * B22
(A21-A11) * (B11+B12)
(A12-A22) * (B21+B22)
Then it is simply a matter of substitution to show that

C11 = P1 + P4 - P5 + P7
C12 = P3 + P5
C21 = P2 + P4
C22 = P1 + P3 - P2 + P6
Instead of 8 multiplications, only 7 are required, at the cost of several more additions.
The reason it does is that we can apply the above formulas recursively. And as we break a matrix of
order N into matrices of order N/2, we have to define 7 values, not 8. But each of those 7 values is also a
matrix multiplication, and hence can be computed by the algorithm, and requires only 7 multiplications,
and not 8. It turns out that the number of quantities we have to define drops precipitously, and so the fact
that we have to use a lot of extra additions to define them doesn't matter.
Strassen's algorithm requires an amount of work that increases with N like NLOG2 7 rather than N3. The
extra additions in the defining formulas cause the work formula to have a larger constant in front of it, so
that for small N, the standard algorithm is faster. But, as we have shown above, there are now
implementations of the Strassen algorithm that beat the best implementations of the standard algorithm
on the Cray.
Reference:
Volker Strassen,
Gaussian Elimination is not Optimal,
Numerische Mathematik,
Volume 13, page 354, 1969.
Submatrix
A submatrix of a matrix A is any rectangular "chunk" of the matrix.
The chosen entries may all be neighbors, or they may be chosen by choosing any subset of the row and
column indices of A, in any order.
As a simple example of the use of submatrices, suppose that A has the form
( A1 A2 )
( 0 A3 )
where A1, A2 and A3 are submatrices. Then a linear system involving A could be solved as two smaller
subproblems: solve A3 * x2 = b2, then solve A1 * x1 = b1 - A2 * x2. ( Here b1 and b2 are the parts of
the right hand side that correspond to the subdivision of A).
Similarly, for this case, the eigenvalues of A can be determined by finding the eigenvalues of A1 and
A3. (The computation of the eigenvectors would be a little more complicated.
Successive Overrelaxation (SOR)

The successive overrelaxation method or SOR is an iterative method for solving linear systems, and is a
generalization of the Gauss Seidel and the Jacobi iterations.
SOR is only appropriate for matrices which are strictly diagonally dominant or else symmetric and
positive definite.
To derive SOR, think of both the Jacobi and Gauss Seidel iterations as computing a correction to the
current estimate of the solution, so that a step of the method has the form:
X[N+1] = X[N] + dX.
SOR offers the ability to add a greater or smaller proportion of the correction, which we will denote w:
X[N+1] = X[N] + w * dX.
Surprisingly, for the appropriate choice of w, the SOR method can converge faster than the Gauss Seidel
or Jacobi methods. It can be shown that the SOR method will only be convergent for 0 < w < 2. Values
of w less than 1 result in an underrelaxed iteration, while values greater than 1 (the usual case)
correspond to an overrelaxed iteration.
For a given coefficient matrix, convergence of the SOR method is optimal for some value of w, but it is
generally not easy to determine this value. A variety of schemes are available for estimating and
adjusting the value used during a particular iteration.
The SOR iteration can be considered in terms of its matrix splitting. That is, if we decompose the matrix
A into its strictly lower triangular, diagonal, and strictly upper triangular parts:
A=L+D+U
xnew = x - w * ( L + D )-1 * ( A * x - b ).
I - w * ( L + D )-1 * A.
If the original coefficient matrix A is symmetric, then it may be preferred to use the symmetric SOR
iteration or SSOR. In this case, the iteration consists of pairs of SOR steps. The odd steps are the same
as the usual iteration. But in the even steps, the variables are solved for in reverse order. Each pair of
such steps is a single step of the SSOR iteration, which has the property that its iteration matrix is
similar to a symmetric matrix (though not necessarily symmetric itself). Among other things, this means
that SSOR can be used as a preconditioner for certain other problems.
Symmetric Matrix
A symmetric matrix A is equal to its transpose, that is,
A = AT
or, for every pair of indices I and J:
A(I,J) = A(J,I)
Every matrix A can be decomposed into the sum of an antisymmetric and a symmetric matrix:
A = B + C = (1/2) * ( ( A - AT ) + ( A + AT ) )
Here is an example of a symmetric matrix:
1
2
0
4
2
9
4
8
0
4
5
3
4
8
3
7
Simple facts about a symmetric matrix A:
A is normal;
the inverse of A is symmetric;
The eigenvalues of A are real;

Eigenvectors associated with distinct eigenvalues are orthogonal to each other;
A has a complete set of eigenvectors, which can be made into an orthonormal set;
A is orthogonally similar to a diagonal matrix.
A has an LDL Factorization.
The eigenvalues of successive members of the sequence of principal minors of a symmetric matrix have
the Sturm sequence property, a strict interlacing relationship. The k+1 eigenvalues of the principal
minor of order k+1 are strictly separated by the k eigenvalues of the minor of order k.
LAPACK, LINPACK and EISPACK include specialized routines for symmetric matrices, and include
the use of symmetric matrix storage to save space.

Symmetric storage is a matrix storage method of storing a symmetric matrix economically, omitting the
repeated elements.
The strict lower triangle of a symmetric or Hermitian matrix is redundant. A symmetric storage scheme
packs the upper triangle of the matrix into a linear vector of length ( N * ( N + 1 ) ) / 2. The data is
organized by columns, with each column starting in row 1 of the original matrix, and proceeding down
to the diagonal.
If A was the matrix:
11
12
13
14
15
12
22
23
24
25
13
23
33
34
35
14
24
34
44
45
15
25
35
45
55
then A could be symmetrically stored as:

1
2
3
4
5
( 11, 12, 22, 13, 23, 33, 14, 24, 34, 44, 15, 25, 35, 45, 55 ).
LAPACK, LINPACK and EISPACK include routines which can operate on data stored in this format.
TESTMATRIX
TESTMATRIX is a collection of subroutines useful for generating and manipulating test matrices.
Many sample matrices are available with known inverse, determinant, eigenvalues, rank, symmetry, and
other properties. These matrices may be used to test software for correctness, or for classroom
demonstrations.
Most of the matrices come from a MATLAB M file collection developed by Nicholas Higham,
Department of Mathematics, University of Manchester, and maintained in the "testmatrix" file
somewhere at the MATLAB web site.
An earlier version of the collection is available, again as MATLAB M files, in ACM TOMS Algorithm
694, in the TOMS directory of the NETLIB web site.
I have a FORTRAN version of the source code available in the TESTMAT page.
Toeplitz Matrix
A Toeplitz matrix is a matrix which is constant along each of its diagonals.
Here is an example of a square Toeplitz matrix:
4
3
2
1
5
4
3
2
6
5
4
3
7
6
5
4
a "wide" rectangular Toeplitz matrix:

3 4 5 6 7
2 3 4 5 6
1 2 3 4 5
a "tall" rectangular Toeplitz matrix:

5
4
3
2
1
6
5
4
3
2
7
6
5
4
3
Facts about a Toeplitz matrix A:
A is persymmetric.
the inverse of A is not, in general, a Toeplitz matrix, but is persymmetric.
Compare the concepts of a Hankel Matrix, a Symmetric Matrix and a Circulant Matrix.
Trace of a Matrix
The trace of a (square) matrix is the sum of the diagonal elements.
Simple facts about the trace of a matrix A:
the trace is equal to the sum of the eigenvalues of A;

if A is similar to B, then trace ( A ) = trace ( B ).
For square matrices A and B, trace ( A * B ) = trace ( B * A ).
The trace of the following matrix is 4:

1 -1 0
-1 2 -1
0 -1 1
and it has three eigenvalues, 0, 1 and 3, whose sum is also 4.

Transpose
The transpose of a matrix A is obtained by switching all pairs of values A(I,J) and A(J,I).
In printed text, the transpose is usually denoted by a superscript T, as in AT, while in running text the
symbols A', A^T, transpose ( A ), or trans ( A ) might be used.
For the square matrix A:
1
4
5
7
2
8
5
0
0
9
6
0
4
2
3
3
the transpose is:

1
2
0
4
4
8
9
2
5
5
6
3
7
0
0
3
For the "wide" rectangular matrix A:

11 12 13 14 15
21 22 23 24 25
31 32 33 34 35
the transpose is the "tall" rectangular matrix:

11
12
13
14
15
21
22
23
24
25
31
32
33
34
35
Simple facts about the transpose of a matrix A:
A is singular if and only if AT is singular;

A and AT have the same determinant;
A and AT have the same characteristic equation;

A and AT have the same eigenvalues;
the left eigenvectors of A are the right eigenvectors of AT, and vice versa;
the transpose of an M by N rectangular matrix is N by M.
the tranpose of a sum is the sum of the transposes:
(A + B)T = AT + BT
the tranpose of a product is the product of the transposes in reverse order:

(A * B)T = BT * AT
The LU factorization of a matrix A allows the solution of linear systems involving AT as well.
LAPACK and LINPACK linear solution software takes advantage of this fact.
Trapezoidal Matrix
A trapezoidal matrix is essentially a rectangular triangular matrix. Thus, a trapezoidal matrix has a
different number of rows than columns. If, in addition, A(I,J) = 0 whenever I > J, the matrix is called
upper trapezoidal. If A(I,J) = 0 whenever I < J, the matrix is called lower trapezoidal.
Here is a "wide" upper trapezoidal matrix:
11 12 13 14
0 22 23 24
0 0 33 34
Here is a "tall" upper trapezoidal matrix:

11 12 13 14
0 22 23 24
0 0 33 34
0 0 0 44
0 0 0 0
0 0 0 0
You might encounter an upper trapezoidal matrix when carrying out Gauss Jordan Elimination on a
square matrix, or computing the QR factorization of a rectangular matrix.

Triangular Matrix
An upper triangular matrix is entirely zero below the main diagonal while a lower triangular matrix is
zero above the main diagonal.
It the entries of the main diagonal are all equal to 1, the matrix is said to be unit upper triangular or unit
lower triangular. For example, you will encounter a unit lower triangular matrix in Gauss elimination.
Simple facts about a triangular matrix A:
The determinant is the product of the diagonal entries;

The eigenvalues of A are the diagonal entries;
The inverse of A is also a triangular matrix;
The linear system A * x = y is very easy to solve;
A is not normal, and hence not diagonalizable, unless it is already actually a diagonal matrix.
Tridiagonal Matrix
A tridiagonal matrix is a matrix whose only nonzero entries occur on the main diagonal or on the two
diagonals which are immediate neighbors of the main diagonal.
The diagonals immediately below and above the main diagonal are referred to as the subdiagonal and
superdiagonal respectively.
Here is an example of a tridiagonal matrix which is also positive definite symmetric:
-2 1 0 0
1 -2 1 0
0 1 -2 1
0 0 1 -2
Simple facts about a tridiagonal matrix A:
A is a band matrix.
Every matrix is similar to a tridiagonal matrix.
A is irreducible if and only if every subdiagonal and superdiagonal element is nonzero.
If Gauss elimination can be performed on A without using pivoting, then the L factor is zero
except for the diagonal and first subdiagonal, and the U factor is zero except for the diagonal and
first superdiagonal.
A-1 is generally not tridiagonal.
If it is true that none of the subdiagonal elements are zero, and none of the superdiagonal elements are
zero, and
|A1,1| > |A1,2|
|An,n| > |An,n-1|
and, for 2 <= i <= n-1,
|Ai,i| >= |Ai,i-1| + |Ai,i-1|
then A is nonsingular, and Gauss elimination can be performed without pivoting. THis is true, for
instance, for the "-1,2,-1" matrix used to approximate the second derivative, and the "2,1;1,4,1;1,2"
matrix used in cubic spline interpolation.
If a (real) matrix A is tridiagonal and irreducible, then its eigenvalues are real and distinct. Moreover,
the eigenvalues of the sequence of principal minors of A have a strict interlacing property: the k+1
eigenvalues of the principal minor of order k+1 are strictly separated by the k eigenvalues of the minor
of order k. Since the determinant of the minors can be easily computed recursively, and since the sign
sequence of these determinants carries information about the number of negative eigenvalues associated
with each minor, this suggests how a bisection method can be employed to hunt for the eigenvalues of a
tridiagonal matrix.
LAPACK and LINPACK include special routines for efficiently solving linear systems with a
tridiagonal coefficient matrix. LAPACK and EISPACK have routines for finding eigenvalues of a
tridiagonal matrix.
Cyclic reduction is a method of solving a tridiagonal system of equations which can give a large speedup
on vector or parallel processors.
Unimodular Matrix
A unimodular matrix is a square matrix whose determinant has absolute value 1.

Facts about a unimodular matrix A:
The inverse matrix A-1 is unimodular;

If A is an integer matrix, then so is A-1;
If B is unimodular, so is A*B;
Examples of unimodular matrices:
the identity matrix;

any permutation matrix;
any orthogonal matrix;
Any diagonal, upper triangular, or lower triangular matrix whose diagonal elements have a
product of +1 or -1.
Unitary Matrix
A unitary matrix is a complex matrix U whose transpose complex conjugate is equal to its inverse:
U-1 = UH
Facts about a unitary matrix U
U * UH = UH * U = I
U is "L2-norm preserving": ||U*x||2 = ||x||2
the columns of U are pairwise orthogonal, and have unit L2-norm.
In real arithmetic, the corresponding concept is an orthogonal matrix.

Unitary Similarity Transformation

A unitary transformation is a relationship between two complex matrices A and B, and a unitary matrix
V, of the form:
A = V-1 * B * V.
Upshift Matrix
The upshift matrix A circularly shifts all vector entries or matrix rows up 1 position.
Example:
0
0
0
1
1
0
0
0
0
1
0
0
0
0
1
0
Facts about the upshift matrix A:
A is a permutation matrix;
A is an N-th root of the identity matrix;
A is persymmetric;
A is a circulant matrix;
the inverse of the upshift matrix is the downshift matrix.
Vector-Bound Matrix Norm

A vector-bound matrix norm is a matrix norm that has been (or can be) derived from a vector norm by
the following formula:
||A|| = supremum ||A*x|| / ||x||
where the supremum (roughly, the "maximum") is taken over all nonzero vectors x.
If such a relationship holds, then expressions involving the matrix norm and vector norm can be mingled
to produce useful inequalities, based on the guaranteed compatiblity relationship:
||A*x|| <= ||A|| * ||x||
Matrix norms which are vector bound with some vector norm include the L1 matrix norm, the L2 matrix
norm, and the L Infinity matrix norm.
Vector Norm
A vector norm is a function ||*|| that measures the "size" of a vector.
A vector norm must have the following properties:
|| V || >= 0, and || V || = 0 if and only if V is the zero vector (positivity);

|| s * V || = | s | * || V || for any scalar s (linearity).
|| V + W || <= || V || + || W || for any vectors V and W (triangle inequality).
Commonly used vector norms include:
The L1 vector norm;

The L2 vector norm;
The L Infinity vector norm.
Given two points in space, x and y, we can define the distance between the points, d(x,y), in terms of a
vector norm operating on the vectors of position coordinates:
d(x,y) = || x - y ||
and this quantity will have the expected properties of a distance function.
For a given vector norm, it is important to know which matrix norms are compatible, so that expressions
like
||A*x|| <= ||A|| * ||x||
may be asserted.
Zero Matrix
The zero matrix is a matrix all of whose entries are zero.

A zero matrix is sometimes called a trivial matrix. A matrix which has at least one nonzero entry is
called a nontrivial matrix or nonzero matrix. (It is not required that all entries be nonzero, just that at
least one of them is nonzero!)
Zero One Matrix

A zero one matrix is a matrix whose entries are equal to 0 or 1.
The adjacency matrix of a graph is a zero one matrix. A permutation matrix is a zero one matrix. Many
combinatorial problems can be formulated in terms of a zero one matrix.
The permanent of a zero one matrix is equal to the number of generalized diagonals that contain no 0
element. The permanent of a zero one matrix is zero if and only if it contains an r by s subblock of
zeroes, with r+s>=n+1.
Back to the home page.

Written by John Burkardt. Last revised on 25 March 2001.
Linear Algebra: Table of Contents
Linear Algebra
Table of Contents
I. System of Linear Equations

System of Linear Equations
Augmented Matrix
Row Operation
Gaussian elimination and back substitution Extra: The three operations do not change solutions
Row operation
Row Echelon Form
Row echelon form
Shape of row echelon form
Computing row echelon form
Reduced row echelon form Extra: Reduced row echelon form vs. General solution
Existence
Existence of solutions for fixed right side
Existence of solutions for any right side
Uniqueness
Structure of solutions
Uniqueness of solutions
A Basic Linear Algebra Principle
Implication of existence/uniqueness on size
Implication of size on existence/uniqueness
II. Linear Transformation and Matrix
Vector and Matrix
Euclidean vector and euclidean space Extra: Vector addition is given by parallelogram
Matrix
Some matrix notations and terminologies
Transformation Extra: Root-Polynomial transformation
Matrix transformation
Linear transformation Extra: Matrix transformations are linear
Linearity in picture
Linear transformation = matrix transformation Extra: Matrix of a linear transformation
Onto
http://algebra.math.ust.hk/course/content.shtml (1 of 6)2005/03/08 04:14:01 .
Range and existence

Onto and always existence
One-to-one
One-to-one and uniqueness
Kernel and uniquess
Composition and Matrix Multiplication
Composition of transformations
Multiplication of matrices Extra: Composition of linear transformations is linear
Properties of matrix multiplication
Composition/multiplication mixed with addition and scalar multiplication
Inverse
Inverse transformation Extra: Criterion for invertibility of a transformation
Inverse matrix
Inverse of 2 by 2 matrix
Computation of inverse matrix
Properties of inverse Extra: Criterion for invertibility of a matrix
Transpose
Partitioned Matrix
Appendix: Transformation
Definition of transformation
Onto
One-to-one
Inverse
Graph of transformation
Formal theory of transformation
III. Vector Space
Vector Space
Motivation
How to make the definition
Definition
Proofs in vector space
Subspace
Definition
Sum and intersection
Definition
Operations of linear transformations and Hom space
Range and Column Space
Range
Column space
Kernel and Null Space
Kernel
Null space
Structure of solutions of general linear equations
Span
Definition and relation to existence
Geometric intuition of span
Properties of span Extra: Properties of span
Linear Independence
Definition and relation to uniqueness
Geometric intuition of linear independence
Properties of independence Extra: Properties of linear independence
Basis
Definition
Coordinate
Isomorphism
Dimension
Properties of dimension
Computation of Basis
Basis of null space
Basis of column space, first method
Basis of column space, second method
Rank and Nullity
Rank and nullity for matrix
Properties of rank and nullity Extra Properties of rank and nullity
Rank and nullity for linear transformation
Change of Basis
Matrix for change of coordinates
Computing the matrix for change of coordinates
Matrix for linear transformation
Change of the matrix for linear transformation
Similar matrix
Dual
Dual space
Dual basis
Dual transformation
Double dual
Direct Sum
Direct sum of vector spaces

Direct sum of subspaces
Properties of direct sum
Direct summand
Linear transformation between direct sums
Appendix: Field
Number system and field
Complex number
Linear algebra over a field*
IV. Determinant
Geometry of Determinant
Determinant of 2 by 2 matrix
Determinant of 3 by 3 matrix
Determinant and volumn change
Computation of Determinant
Row/Column operations
Cofactor expansion
Combination of two methods
Properties of Determinant
Properties
Invertibility and determinant
Cramer's rule
Theory of Determinant
Definition of determinant
General formula
Existence and uniqueness
Generalization
Proof of Properties
Transpose
Justification of computation
Justification of geometry
V. Eigenvalue and Eigenvector
Eigenvalue and Eigenvector
Definition
Computation
Diagonalization
Complex Diagonalization
Complex eigenvalue and eigenvector
Complex eigenvalue and eigenvector for real matrix
Complex eigenvalue in real diagonalization
Application
Discrete dynamical system
Functions of matrix
Linear differential system
Properties
About single matrix
About several matrices
About polynomials of matrix
Trace and determinant
Eigenspace and Multiplicity
Direct sum of eigenspaces
Algebraic and geometric multiplicities
Multiplicity and diagonalization
Polynomials of Matrix
Cayley-Hamilton theorem
Minimal polynomial
Multiplicity and diagonalization
Appendix: Polynomial
Root and multiplicity
Representing polynomial by roots
Division of polynomials
Greatest common divisor
VI. Inner Product
Inner Product Space
n
Dot product and euclidean geometry of R
Inner product
n
Inner product on R
Duality and Adjoint
Orthogonality
Orthogonal and orthonormal
Properties of orthogonality
Orthonormal basis and isometry
Orthogonal Projection
Orthogonal projection
Gram-Schmidt process
Orthogonal Complement
Definition and properties
Complementarity principle
Complex Inner Product
Definition and properties
Complementarity principle
Last Updated: 3 December 2002

Copyright 2002 by Min Yan. All rights reserved.
Department of Mathematics, Hong Kong University of Science and
Technology, Clear Water Bay, Hong Kong
Linear Algebra
Langara College - Department of Mathematics and Statistics
Internet Resources for College Math Students
Linear Algebra
Systems of Equations, Vector Geometry, Matrix Algebra, Linear Transformations,
Numerical and Computational Issues, Applications
FullCourses and CrossTopicProjects

ubcM152
Elementary Linear Algebra(by [email protected])
Linear Algebra WebNotes(byMarkSapir@vanderbilt)
Lecture Notes from a Linear Algebra Course at Cornell(byRLance)
Elements of Abstract and Linear Algebra(byEConnell@uMiami)
MultivarCalc, LinAlg, & DiffEq(inCCPbyWattenberg@montana)
IB Matrices and Vectors (@CIS in HK)
vectorjava@ies, miscjava@ies
Individual Topic Items

Systems of Equations
http://www.hofstra.edu/~matscw/tutorialsf1/frames2_2.htmlfor 2 unknowns
http://www.hofstra.edu/~matscw/tutorialsf1/frames2_3.html for 3 unknowns
2x2systems@hofstra
3x3systems@hofstra
3x3 System by Spreadsheet(@ArkansasStU)
Gaussian Elimination@OregonState
review of linear systems @bcit
Cramer's Rule@ies
Vector Geometry (2&3d)

http://www.langara.bc.ca/mathstats/resource/onWeb/linalg/ (1 of 3)2005/03/08 04:14:11 .
Linear Algebra
Vectors -byGeneKlotz@MathForum(uses Geometer'sSketchpad)

Definition and General Properties of Vectors@polytechU(penn)
Graphical Illustration of Vector Addition@polytechU(penn)
Graphing Vector Calculator - by Paul Flavin
Vector Sums Applet - by Dale Stevenson
Vector and field(1)@ies
Vector and field(2)@ies
Decomposing a vector in its rectangular component form@polytechU(penn)
Graphical Illustration of the Dot Product for Vectors@polytechU(penn)
dotProduct@ies
MathsOnlineGallery (@uVienn.austria) - 3dVectors
MathsOnlineGallery (@uVienn.austria) 3dCoords&Lines
MathsOnlineGallery (@uVienn.austria) 3dPlanes
This cross product demo has, I think, a very natural user interface.
cross product demo(3dpic)byPhys@syracuse
The Cross Product @swarthmore
The Cross Product@OregonState
Vector Cross Product - JAVA Interactive Tutorial
Distributivity of CrossProduct (@ies)
Interior and exterior division vector@ies
Equations of Lines and Planes@OregonState
Planes@CRCviaGeomCentre@umn
Matrix Algebra
Matrices@OregonState
Determinants@OregonState
S.O.S. Math - Matrix Algebra
Mickey's Linear Systems Materials
Linear Transformation of Lines@ies
Linear Transformation of Shape@ies
Numerical and Computational Issues

Numerical Linear Algebra
Linear Algebra
Linear Algebra Computer Exercises
Applications
Application linear equation systems to surveying(by eh@bcit)
What do I do? My business depends on this!(by eh@bcit)
Applications of Linear Algebra and Matrices to Electronics(by eh@bcit)
An application of linear algebra and matrices in chemistry(by eh@bcit)
Stability of Structures(by eh@bcit)
The Great Balsa Bridge Contest@uHawaii
Viewing Objects in Computer Graphics - An Application of Matrices(by eh@bcit)
If you have come across any good web-based materials on linear algebra that are not
included in the above indexes, please do let us know and we may add them here.
Back to Math&Stats Dep't Homepage
.......
Give
Return to Langara College
.......
Feedback
Homepage
Linear Algebra With Applications
Linear Algebra
With
Applications
Sixth Edition
by
Steven J. Leon
Important Note: To use our
websites, we recommend
that you use version 5 or
greater of Microsoft Internet
Explorer or version 4 or
greater of Netscape
Navigator. In both cases, make sure you have JavaScript,
cookies, and Java enabled.
If you need to upgrade your browser, go to our Browser Tuneup page.
Select a section:
Section 1.1: Systems of Linear Equations
1999-2000 by Prentice-Hall,
Inc.
A Pearson Company
Distance Learning at Prentice Hall
Legal Notice
http://cwx.prenhall.com/bookbind/pubbooks/esm_leon_linear_6/2005/03/08 04:14:45 .
Main Page
Calculus
Calculus, 3-Dimensional
Enter the Matrices you want to work with here:
A=
b=
Differential Equations
Financial
Graph Theory
Check which operations you would like to perform:
Linear Algebra
Numerical Methods
Statistics, General
Statistical Regression
Invert A.
Transpose A.
Row Reduce A.
Find the determinant of A.
Find the eigenvalues of A.
Find the eigenvectors of A.
Solve Ax=b for x.

Expert
Compute
Undo Please allow up to 1 minute.
If you have comments or questions about this page, please email Dr. Robert H. Hoar or Matt Bedalov.
http://www.compute.uwlax.edu/lin_alg/2005/03/08 04:15:33 .
IIC - paper journals
Information about Paper Journals
Linear Algebra and its Applications (LAA)
LAA Homepage
Contents of LAA, years 1991-1999
Contents of LAA, year 2000
Special issues of LAA in progress
Special issues of LAA in print
LAA Referee's evaluation form (LaTeX file)
Elsevier TeX Styles' directory
Linear and Mulitinear Algebra (LAMA)
LAMA Homepage
http://www.math.technion.ac.il/iic/journals.html (1 of 2)2005/03/08 04:15:39 .
IIC - paper journals
Numerical Linear Algebra with Applications (NLAA)
NLAA Homepage
SIAM Journal on Matrix Analysis and Applications (SIMAX)
SIMAX homepage
Contents of SIMAX, years 1991-1998
Mathematics of Control, Signals, and Systems (MCSS)
MCSS HomePage
MCSS HomePage (Rutgers Copy)
Numerical Mathematics (BIT)
Contents of NLAA, years 1992-today
BIT homepage
Journal of Computational Analysis and Applications (JoCAAA)
JoCAAA homepage
Back to ILAS information center

http://www.math.technion.ac.il/iic/journals.html (2 of 2)2005/03/08 04:15:39 .
> M:=matrix([[22,-128,56,56,-48],[-10,48,-21,-7,-3],[0,70,7,-35,35],[-34,138,-105,-21,1],[-14,28,-28,0,0]]);
> ei:=[eigenvects(M)];
> v1:=ei[1][3][1];
v2:=ei[1][3][2];
v3:=ei[2][3][1];
v4:=ei[2][3][2];
v5:=ei[3][3][1];
> P:=augment(v1,v2,v3,v4,v5);
> evalm(inverse(P)&*M&*P);
> N:=matrix([[14,-112,40,56,-56],[-7,42,-15,-7,0],[7,56,7,35,42],[-35,140,-107,-21,0],[-14,28,-28,0,0]]);
> ei:=[eigenvects(N)];
> v1:=ei[1][3][1];
v2:=ei[2][3][1];
v3:=ei[2][3][2];
v4:=ei[3][3][1];
> Q:=augment(v1,v2,v3,v4,id(5));
> rref(Q);
> P:=submatrix(Q,1..5,1..5);
> evalm(inverse(P)&*N&*P);
Note that the eigenvalue l = 28 is the one eigenvalue whose geometric multiplicity is not
equal to its algebraic multiplicity (which is 2). Consider (28*I - N)2, where I have raised
28I - N to the algebraic multiplicity of 28.
Computing the null space of this matrix, we now get two vectors.
> ge28:=nullspace(evalm((28*id(5)-N)^2));
> w1:=ge28[1];w2:=ge28[2];
> P:=augment(v1,v2,v3,w1,w2);
> evalm(inverse(P)&*N&*P);
This is the most "simple" matrix that is similar to N.
ITERATIVE METHODS FOR COMPUTING

EIGENVALUES
The Power Method

> A:=Matrix([[-726, -184, -280], [2708, 687, 1045], [100, 25, 51]]);
-184
-280
-726
A := 2708
687
1045
25
51
100
> x[0]:=<1.,0,0>;y[0]:=x[0];
1.

x0 := 0

0
1.

y0 := 0

0
> n:=500;
n := 500
> for j from 1 to n do
x[j]:=A.y[j-1]:
m[j]:=Norm(x[j],infinity);
y[j]:=x[j]/m[j]: od:
> [(x[n][1]/y[n-1][1]),(x[n][2]/y[n-1][2]),(x[n][3]/y[n-1][3])];
[26.00000000, 26.00000001, 25.99999999]
> y[n];
-0.260869565162612616
0.99999999979001208
0.0434782608604351515
This tells us that 26 is the dominant eigenvalue of A.
> A.y[n];
-6.78260869422730561
25.9999999945380402
1.13043478237123440
Note that this implies y[n] is approximately an eigenvector of A for the eigenvalue of 26.
Using the Rayleigh Quotient

> for j from 1 to n do
x[j]:=A.y[j-1]:
m[j]:=DotProduct((A.x[j]),x[j])/DotProduct(x[j],x[j]);
y[j]:=x[j]/m[j]: od:
> [seq(m[j],j=1..200)];
[-0.4542370469, -1113.513250, 1.127040640, 534.5507064, 3.066061949, 204.4052228, 5.029245549,
125.8558539, 6.948773613, 91.72714738, 8.797417318, 72.89777208, 10.55625738,
61.09532504, 12.21087445, 53.09265521, 13.75104972, 47.37274919, 15.17063737,
43.12857623, 16.46724185, 39.89114197, 17.64172455, 37.36918796, 18.69761291,
35.37223050, 19.64048015, 33.77042443, 20.47735009, 32.47221394, 21.21616324,
31.41120449, 21.86532509, 30.53810178, 22.43334347, 29.81557221, 22.92855245,
29.21485831, 23.35891456, 28.71348637, 23.73188935, 28.29367485, 24.05435591,
27.94120468, 24.33257726, 27.64460169, 24.57219558, 27.39453403, 24.77824959,
27.18336107, 24.95520633, 27.00479081, 25.10700197, 26.85361626, 25.23708718,
26.72550990, 25.34847392, 26.61686225, 25.44378164, 26.52465317, 25.52528147,
26.44634912, 25.59493734, 26.37981971, 25.65444384, 26.32327019, 25.70526037,
26.27518603, 25.74864179, 26.23428736, 25.78566560, 26.19949122, 25.81725602,
26.16988046, 25.84420494, 26.14467751, 25.86719034, 26.12322279, 25.88679227,
26.10495630, 25.90350669, 26.08940248, 25.91775737, 26.07615711, 25.92990637,
26.06487665, 25.94026288, 26.05526892, 25.94909073, 26.04708535, 25.95661514,
26.04011447, 25.96302830, 26.03417632, 25.96849404, 26.02911772, 25.97315217,
26.02480821, 25.97712192, 26.02113681, 25.98050495, 26.01800895, 25.98338787,
26.01534408, 25.98584460, 26.01307366, 25.98793807, 26.01113925, 25.98972202,
26.00949110, 25.99124217, 26.00808688, 25.99253751, 26.00689043, 25.99364129,
26.00587102, 25.99458182, 26.00500243, 25.99538327, 26.00426238, 25.99606616,
26.00363180, 25.99664804, 26.00309452, 25.99714387, 26.00263672, 25.99756635,
26.00224665, 25.99792634, 26.00191431, 25.99823309, 26.00163110, 25.99849446,
26.00138981, 25.99871717, 26.00118420, 25.99890693, 26.00100903, 25.99906862,
26.00085977, 25.99920641, 26.00073257, 25.99932380, 26.00062420, 25.99942383,
26.00053187, 25.99950906, 26.00045318, 25.99958168, 26.00038614, 25.99964355,
26.00032902, 25.99969629, 26.00028035, 25.99974122, 26.00023888, 25.99977949,
26.00020355, 25.99981210, 26.00017343, 25.99983991, 26.00014778, 25.99986360,
26.00012591, 25.99988377, 26.00010729, 25.99990096, 26.00009143, 25.99991562,
26.00007791, 25.99992810, 26.00006636, 25.99993873, 26.00005656, 25.99994778,
26.00004819, 25.99995551, 26.00004106, 25.99996209, 26.00003499, 25.99996771,
26.00002980, 25.99997247, 26.00002540, 25.99997656, 26.00002164, 25.99998003,
26.00001845, 25.99998298, 26.00001572, 25.99998549, 26.00001338, 25.99998765,
26.00001140, 25.99998946, 26.00000972, 25.99999103, 26.00000828, 25.99999235,
26.00000707]
This says the Rayleigh Quotient approximates the value of the dominant eigenvalue.
A Procedure
> itv:=proc(A,v,n) local x,y,j,m,k,i;
k:=Dimension(v);
x[0]:=v;
y[0]:=v;
for j from 1 to n do
x[j]:=A.y[j-1]:
m[j]:=Norm(x[j],infinity);
y[j]:=x[j]/m[j]:
od;
print(seq((x[n][i]/y[n-1][i]),i=1..k));
y[n];
end;
itv := proc(A, v, n)
local x, y, j, m, k, i;
k := (LinearAlgebra:-Dimension)(v );
x[0] := v;
y[0] := v;
for j to n do x[j] := `.`(A, y[j - 1]);
m[j] := (LinearAlgebra:-Norm)(x[j], infinity);
y[j] := (x[j])/(m[j]);
end do;
print(seq((x[n][i])/(y[n - 1][i]), i = 1 .. k ));
y[n];
end proc;
> itv(A,<1.,0,0>,100);
25.97588703, 25.97719692, 25.98191308
-0.260863248135825510
1.00000000000940248
0.0434820511203855426
The Shifted Power Method

> B:=A-26*IdentityMatrix(3);
-752
B := 2708
100
-184
661
25
-280
1045
25
> itv(B,<1.,0,0>,100);
-50.00000002, -50.00000000, -50.00000000
0.275862068944673688
-0.99999999992444188
-0.0344827586180841486
This gives us that -50 is an eigenvalue of B or -50 + 26 = -24 is an eigenvalue of A.
> C:=B + 50*IdentityMatrix(3);

-702
C := 2708
100
-184
711
25
-280
1045
75
> itv(C,<1.,0,0>,100);
50.00000000, 50.00000000, 49.99999999
-0.260869565196463038
0.99999999991977618
0.0434782608660772841
This tells us that 50 is an eigenvalue of C or that 50 - 50 = 0 is an eigenvalue of B or that 0 + 26 = 26 is an
eigenvalue of A. We already knew this.
Modification
We know 26 is an eigenvalue of A. Lets find the corresponding eigenspace.
> eig:=NullSpace((26*IdentityMatrix(3)-A));
-6

eig := 23

1
> v1:=eig[1];
-6

v1 := 23

1
T
Thus the eigenspace is spanned by the vector [-6, 23, 1] . Find vectors orthogonal to this.
> M:=<v1>;nul:=NullSpace(Transpose(M));
-6

M := 23

1
nul :=
23
,
1
> v2:=nul[1];v3:=nul[2];
23
v2 :=
1
v3 :=
> P:=<v1|v2|v3>;
-6
P :=
23
23
6
1
0
> Q:=P^(-1).A.P;
26
Q := 0
409150
849
-4667
283
-20825
283
55334
849
-765
283
705
283
> E:=SubMatrix(Q,2..3,2..3);
-4667
283
E :=
-20825
283
-765
283
705
283
> itv(E,<1.,0>,100);
-24.00000000, -24.00000000
0.360000000030629040
1.00000000008508082
> eig2:=NullSpace(-24*IdentityMatrix(3)-A);
-8

eig2 := 29

1
> v2:=op(1,eig2);
-8

v2 := 29

1
> M:=<v1|v2>;nul2:=NullSpace(Transpose(M));
-8
-6
M := 23
29
1
1
nul2 :=
-3
-1
> v3:=nul2[1];
v3 :=
> P:=<v1|v2|v3>;
-3
-1

-6
P :=
23
-8
29
1
-3
-1
> F:=P^(-1).A.P;
26
F :=
0
0
We have thus found all three eigenvalues for A.
-24
0
16
-136
10
GAUSSIAN ELIMINATION
LAB #1
Throughout this lab, you should execute all of the examples in Maple and briefly examine the results.
Beginning a Maple Session in Linear Algebra

ENTERING MATRICES
At this point we will mention three different ways to enter a matrix
in Maple. Maple uses the programming convention that matrices
are special data constructs. Consequently, all three of these ways
use the matrix function call.
Method 1
2
If you are given a particular matrix, for example A = 4
-1 , you can enter it rowwise by simply
entering:
> A:=Matrix([[2,8],[4,-1],[3,5]]);
or you can define it columnwise by entering:
> A:=<<2,4,3>|<8,-1,5>>;
Note that to enter the matrix you simply type the row entries separated by commas and enclose each of
those in rectangular brackets separated by commas. The list of rows is also then enclosed in rectangular
brackets. The entries that you put in the rows can be almost anything you want. They can be letters
representing numbers, functions, or any type of Maple expression. Thus for example we could enter
2
x
x
the matrix B =
into Maple by typing:
3
4
x
x
> B:=<<x,x^3>|<x^2,x^4>>;
This will be the method most commonly used in this class to enter a matrix.
Method 2
This method can be used when you want to define a matrix within unknown
entries, but having a particular number of row and columns. For example, if you
wanted C to be an arbitrary 3x2 matrix, you could use the Maple command:
> C:=Matrix(3,2,symbol=c);
Note that the letter "c" cannot be previously defined as something else. To see a matrix at any time,
you can type:
> A,B,C;
Method 3
The third method will not be used much but defines matrices whose entries are a
function of the row and column number (i,j) (i=row number,j=column number).
If you wanted to define a 3x4 matrix H where the (i,j) entry of the matrix was
2i+j you could issue the following two commands with the corresponding
responses from Maple:
> f:=(i,j)->2*i+j;
> H:=Matrix(3,4,f);
EDITING MATRICES
The i^th row and j^th column entry of a matrix A is denoted by
A[i,j]. For example, if we now type the following we get:
> A;A[1,2];
While it is possible to edit matrices by altering the syntax of the matrix command directly, another, and
sometimes more efficient, means of editing a matrix is by altering particular entries directly using:
A[i,j]:=expr
where expr is any expression. For example, the following commands:
> A[1,2]:=0;
> A[2,1]:=0;
> A[3,2]:=0;
would alter matrix A so that A = 0
-1 . To see the altered version of A simply type:
> A;
ELEMENTARY ROW OPERATIONS

The three elementary row operations for a matrix are represented
in Maple by the commands:
RowOperation(A,m,expr)
RowOperation(A,[m,n],expr)
th
th
This command interchanges the m and n row of a matrix A. Thus, to perform

the row operation that switches rows R1 and R2 on the modified matrix A, one
must enter
> RowOperation(A,[1,2]);
Rowoperation(A,m,expr)
This command multiplies the m^th row of a matrix A by some the expression
expr. To perform the row operation, 3 R1, one must enter
> RowOperation(A,1,3);
One should note that the swaprow command issued in the command previous to this last one did not
change the matrix A. It only gave an output matrix which had the two rows of A interchanged. If we
had made an assignment of this to the variable A, then A would be changed.
RowOperation(A,[n,m],expr)
This command adds the product of the expression expr (most often a scalar) and
th
th
the m row of a matrix A to the n row of Matrix A. To perform the row
operation, 5 R2 + R3, one must type
> RowOperation(A,[3,2],5);
EXAMPLES
Example 1
sin(x)
Enter the matrix C = cos(x)
tan (x)
Solution
> C:=<<sin(x),cos(x),tan(x)>>;
Example 2
0
Enter the matrix E = -2
3
4
1
-2 . Then find the reduced row echelon form of E.
Solution
> E:=Matrix([[0,3,6],[-2,4,-2],[4,1,5]]);
> RowOperation(E,[1,2]);
> RowOperation(%,1,-1/2);
> RowOperation(%,[3,1],-4);
> RowOperation(%,2,1/3);
> RowOperation(%,3,-1/17);
> RowOperation(%,[1,2],2);
This last matrix shows that the reduced row echelon form of E is the 3x3 identity matrix.
It may be to your benefit to name each of your resulting matrices in some way, for if you make an
error, the call for the previous computation (%) will always give you the matrix resulting from the
error. Thus as a precaution, you can name each of your matrices. For example, with the above
example you could have typed:
> E:=Matrix([[0,3,6],[-2,4,-2],[4,1,5]]);
> E1:=RowOperation(E,[1,2]);
> E2:=RowOperation(E1,1,-1/2);
> E3:=RowOperation(E2,[3,1],-4);
> E4:=RowOperation(E3,2,1/3);
> E6:=RowOperation(E5,3,-1/17);
> E9:=RowOperation(E8,[1,2],2);
Example 3
Solve the following system of linear equations.
-x2 - x3 + x4 = 0
x1 + x2 + x3 + x4 = 6
2 x1 + 4 x2 + x3 - 2 x4 = -1
3 x1 + x2 - 2 x3 + 2 x4 = 3
Solution
We first set up the following 4 x 5 augmented matrix M=
[2, 4, 1, -2, -1], [3, 1, -2, 2, 3]]
[[0, -1, -1, 1, 0], [1, 1, 1, 1, 6],
> M:=<<0,1,2,3>|<-1,1,4,1>|<-1,1,1,-2>|<1,1,-2,2>|<0,6,-1,3>>;
Note that even though Maple will not show the vertical line between columns four and five denoting
an augmented matrix,we can reduce the matrix to reduced row echelon form.
> M1:=RowOperation(M,[1,2]);
> M2:=RowOperation(M1,[3,1],-2);
> M4:=RowOperation(M3,2,-1);
> M6:=RowOperation(M5,[4,2],2);
> M7:=RowOperation(M6,3,-1/3);
> M9:=RowOperation(M8,4,-1);
> M12:=RowOperation(M11,[3,4],-2/3);
>
The resulting matrix shows that x1= 2, x2 = -1, x3 = 3, and x4= 2 is the only solution.
PROBLEMS
Turn in a print out of your work (not of the examples). Label
problems clearly. Remember to save your work fairly regularly (z
S), so that if a system error does occur, most of your work will not
be lost. You may want to clear Maple variables after each problem
using the restart command.
Problem 1
Enter the matrix F=
-2
-1
-1
Perform the row operation indicated in a) on F using Maple. Then perform the row
operations indicated in the remaining parts on the matrix resulting from the previous part.
a)
-2 R1 + R2
b)
5 R3
c)
R2 < - > R3
d)
R4 < - > R1
e)
10 R1
f)
79 R4 < - > R2 , (Hint: two steps necessary)
Problem 2
Solve the following system of equations using Maple.
x2 + x3 + x4 = 0
3 x1 + 3 x3 - 4 x4 = 7
x1 + x2 + x3 + 2 x4 = 6
2 x1 + 3 x2 + x3 + 3 x4 = 6
Problem 3
3
If the augmented matrix for a system of equations is 4
8 , use Maple to find the
reduced row echelon form of this matrix to solve the system.
Problem 4
For the following systems, use Gaussian elimination to obtain an equivalent system whose coefficient
matrix is in reduced row echelon form. If the system is consistent, find the solution(s).
a)
-8 x1 - 9 x2 = 5
-2 x1 - 4 x2 = -1
9 x1 - 6 x2 = 6
b)
x1 + 3 x2 + x3 + x4 = 3
2 x1 - 2 x2 + x3 + 2 x4 = 8
x1 + 11 x2 + 2 x3 + x4 = 6
Problem 5
Consider a system whose augmented matrix is of the form 1
a)
For what values of a and b will the system have infinitely many solutions?
b)
For what values of a and b will the system be inconsistent?
MATRIX ARITHMETIC
LAB #2
Throughout this lab, you should type all of the examples into Maple and briefly examine the results.
Arithmetic Commands
The three basic arithmetic operations one can perform on matrices are scalar multiplication, addition, and
multiplication. There are two ways to accomplish each of these in Maple. The two ways for each operation
are respectively:
ScalarMultiply(A,expr) or expr*A
This command takes a matrix and an expression,respectively, and multiplies every entry of the matrix A
by the expression expr.
> A:=<<1,2,3>|<4,5,6>|<7,8,9>>;
> B:=<<-7,-8,4>|<-2,-3,-1>|<8,-5,5>>;
> ScalarMultiply(A,2);
> 2*A;
> Pi*B;
> ScalarMultiply(B,Pi);
Add(A,B) or A+B
This command sums the two matrices A and B which must have the same number of rows and columns.
> Add(A,B);
> A+B;
Multiply(A,B) or A.B
This command multiplies the two matrices A and B for which the number of columns in A is equal to the
number of rows in B.
> Multiply(A,B);
> A.B;
> Multiply(B,A);
> B.A;
Note that AB BA. The operation A.B is used to let Maple know that the multiplication of these matrices
is not necessarily communtative.
Command Names versus Operators

Note that the use of the operator symbols of +,*,-, and . are versatile and can handle all three of the basic
arithmetic operations. These are the easiest to use because their syntax is so similar to what we would
ordinarly write. I would recommend using these rather than ScalarMultiply, Add, or Multiply. They
are more natural and easier to use. After all, if you wanted to add three matrices you would either do:
> A+B+C;
or
>Add(A,Add(B,C));
I find the use of the operator symbols easier.
For example,
> 3*A-4*A.B+5*A^2;
2
computes 3 A - 4 A B + 5 A .
Complex arithmetic expressions are most easily evaluated using these operator symbols. For example,
the following two commands accomplish the same thing but the use of evalm is much easier to input and
identify what it accomplishes.
> 2*B+A.B+B^3;
> Add(Add(ScalarMultiply(B,2),Multiply(A,B)),Multiply(B,Multiply(B,B)))
;
expr+A
This command adds the expression expr to each diagonal entry of the matrix A. We note this means that
if A is a 3x3 square matrix then expr really represents expr*I where I is the 3x3 identity matrix.
> A;
> 2+A;
> B;
> Pi+B;
A^n
(-1) m
n
This computes A where n is an integer. If m is a positive integer, then A^(-m) computes (A ) .
(-1)
Another way to say this is that A^(-1) computes the multiplicative inverse of A if it exists. If A
does
not exist, then Maple will tell you the matrix is singular and that the computation cannot be completed.
> A^3;
> A.A.A;
> A^(-1);
> B^(-1);
> %.B;
Creating Identity and Random Matrices

IdentityMatrix(n)
This command creates and nxn identity matrix and this can be used whenever you need an nxn identity
matrix.
> IdentityMatrix(2);
> B.B^(-1)-IdentityMatrix(3);
> 3*IdentityMatrix(2);
randmatrix(m,n)
This allows you to create a matrix with randomly chosen integer entries in the range from -99 to 99. The
integer m is the number of rows, and n is the number of columns.
> RandomMatrix(1,3);
Note that a different matrix is obtained each time the RandomMatrix command is used.
RandomMatrix(m,n,generator=lowint..highint)
This allows you to create an mxn matrix with randomly chosen integer entries in the range of lowint to
highint. Again m represents the number of rows and n the number of columns.
> M:=RandomMatrix(3,4,generator=-4..7);
The Transpose and Inverse of a Matrix

Transpose(A)
Issuing this command results in output which is the transpose of the matrix A.
> A:=RandomMatrix(2,4,generator=-9..9);
> B:=Transpose(A);
> Transpose(B)-A;
MatrixInverse(S)
The command gives as output the inverse of the square matrix S if it exists. Otherwise it tells you the
matrix S is singular (noninvertible).
> S:=RandomMatrix(2,2,generator=-5..5);
> T:=MatrixInverse(S);
> S.T;
> T.S;
> M:=<<1,4,7>|<2,5,8>|<3,6,9>>;
> MatrixInverse(M);
Matrix Operations (revisited)

The MatrixInverse and Transpose commands can be used in conjunction with operator symbols.
> MatrixInverse(T)+Transpose(S);
Note that one can verify some identities for nxn matrices where n is fixed. For example, let us verify that
(A
(-1) T
T (-1)
) = (A )
for an arbitrary 2x2 invertible matrix A.
> A:=<<a,b>|<c,d>>;
> Transpose(MatrixInverse(A));
> MatrixInverse(Transpose(A));
We observe that the above two matrices are equal, and hence the desired identity is established for 2x2
matrices. Of course the equality of the above two matrices could have been more simply observed by
noting that the following gives:
> Transpose(A^(-1))-(Transpose(A))^(-1);
Finally, note that this does not prove the identity, in general, for nxn matrices. We could use Maple to
verify the identity for 3x3 matrices, then 4x4 matrices, then 5x5 matrices, etc. But we would never finish
this program in our lifetime. We must have some other approach to verify the identity in general.
Help
You can get help on most linear algebra commands by simply typing in general, help(LinearAlgebra[
command name]). For example:
> help(LinearAlgebra[RandomMatrix]);
>
For help, in general, on the linear algebra package go to Linear Algebra.
Matrix Arithmetic Problems

Turn in a printout of your work (not of the examples). Label problems clearly.
Remember to save your work fairly regularly (z S), so that if a system error does
occur, most of your work will not be lost. You may want to clear Maple variables
after each problem (restart).
Problem 1
-7
Let A = -2
8
a) 2A
b) A+B
-8
-3
-5
4
3
-1 and B = -9
5
-2
8
7
5
-9
-3 . Compute the following:
c) 3A - 4B
T
d) (2 A) - (3 B)
e) AB
f) BA
T
g) A B
h) (B A)
Problem 2
2
Let C = -5
-1
-5 and D =
-2
3
. Verify that:
-4
a) 2(CD)= (2C)D = C(2D)

T
b) (C D) = D C
Problem 3
a
Let F =
c
e
b
,G=
g
d
a) (F + G) + H = F + (G + H)
b) (FG)H = F(GH)
c) F(G + H) = FG + FH
d) (G + H)F = GF + HF
i
f
, and H =
k
h
j
. Verify that:
l
e) (F G H) = H G F
f) State what conditions would need to hold in order for FG = GF to be true.
Problem 4
Let J =
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
2
1
-
2
1
-
2
.
1
-
2
a) Compute J .
b) Compute J .
c) When n is a positive integer, what will J
(2 n)
and J
(2 n + 1)
be.
Problem 5
1
The 2x2 matrix K =
-1
1
2 0
has the property that K =
0
-1
0
.
0
a) Is it possible for a symmetric 2x2 matrix to have this property? Why or why not?
a
b) For an arbitrary 2x2 matrix K =
c
Problem 6
b
2 0
what must be true in order for K =
0
d
0
?
0
Find the inverse of each of the following matrices.

-1
a)
1
2
b)
3
c) 0
-1
d) 2
-3
1
0
-3
6
8
Problem 7
Let L =
a) Find L
b) Use L
(-1)
(-1)
to solve Lx = b for the following choices of b.
i)
b =
ii)
b =
iii)
b =
-2
Problem 8
5
Let M=
3
6
3
,N=
2
2
4
2
, and P =
-6
4
Solve each of the following matrix equations for X.

a) MX + N = P
b) XM + N = P
-2
.
3

David Pool

Uploaded by

David Pool

Uploaded by

Linear Algebra

Linear algebra is one of the most useful courses a student of science or

David Poole, LINEAR ALGEBRA: A Modern Introduction

November 4: Draft 1 of Proof 3

http://www.wooster.edu/math/linalg/ (1 of 3)2005/03/08 03:48:53 .

Tentative Lecture Schedule

Short History of Linear Algebra

http://www.wooster.edu/math/linalg/ (2 of 3)2005/03/08 03:48:53 .

Linear Algebra by Jim Hefferon

The College of Wooster Home Page

Last Updated: July 27, 2004

http://www.wooster.edu/math/linalg/ (3 of 3)2005/03/08 03:48:53 .

Mathematics and Computer

Mathematics and Computer Science

2005 Summer Institute for AP Calculus

http://www.wooster.edu/math/jhartman.html (1 of 3)2005/03/08 03:50:58 .

2005 SPRING SEMESTER COURSES

Math 112 - Section 2

Calculus and Analytic Geometry II

Senior Independent Study

Antoney Calistes - Differential Geometry

http://www.wooster.edu/math/jhartman.html (2 of 3)2005/03/08 03:50:58 .

Applications, 140: 267-278 (1990).

Last updated: 11 February 2005

http://www.wooster.edu/math/jhartman.html (3 of 3)2005/03/08 03:50:58 .

The College of Wooster

Campus Tour | Interactive | Wooster Webcam

Popular Wooster Sites

As I have gotten to know

About the College

College meets Walton Foundation challenge, then raises $2

The College of Wooster 2005

To learn about matrices and their properties.

http://www.wooster.edu/math/linalg/policies.html (2 of 3)2005/03/08 03:52:14 .

Math 211 Home Page

http://www.wooster.edu/math/linalg/policies.html (3 of 3)2005/03/08 03:52:14 .

TENTATIVE LECTURE SCHEDULE

http://www.wooster.edu/math/linalg/lecture.html (1 of 2)2005/03/08 03:52:27 .

Return to Math 211 Home Page

http://www.wooster.edu/math/linalg/lecture.html (2 of 2)2005/03/08 03:52:27 .

A Brief History of Linear Algebra

Determinants Used Before Matrices

Implicit Use of Matrices

Late 18th century

http://www.wooster.edu/math/linalg/history.html (1 of 3)2005/03/08 03:52:33 .

Linear Algebra in the Curriculum

Birkhoff and MacLane

http://www.wooster.edu/math/linalg/history.html (2 of 3)2005/03/08 03:52:33 .

Math 211 Home Page

http://www.wooster.edu/math/linalg/history.html (3 of 3)2005/03/08 03:52:33 .

http://www.wooster.edu/math/linalg/writeassign.html (1 of 3)2005/03/08 03:52:45 .

http://www.wooster.edu/math/linalg/writeassign.html (2 of 3)2005/03/08 03:52:45 .

First part of the semester

December 10, 2004

http://www.wooster.edu/math/linalg/writeassign.html (3 of 3)2005/03/08 03:52:45 .

Build an Online Photo Album

Matrix Calculator - NEW VERSION AVAILABLE!

http://www.angelfire.com/space/matrixcalc/ (1 of 2)2005/03/08 03:54:11 .

Variables are assigned using = as in the following example:

http://www.angelfire.com/space/matrixcalc/ (2 of 2)2005/03/08 03:54:11 .

There will be a leading 1 in the last column of G, i.e. in d.

Express any columns of M, that are possible, as linear combinations of other

Find a spanning set for the homogeneous system of linear equations

5.(10 PTS) Suppose u, v, and w are nonzero orthogonal vectors.

Show that u, v, w are linearly independent.

Show that if is a scalar and y is in the range of A then y is in the range of

Solutions for this part are attached.

Show that the 3 vectors above are linearly independent.

What conditions must be placed upon an mxn matrix M to guarantee that it

A matrix A is said to have a right inverse if there is a matrix B so that AB=I.

Thus the matrix A above is a left inverse for M.

x6 = x6, x7 = x7, x8 = x8, x9 = x9

the matrix above is a particular second left inverse for M.

The leading 1 in the last column verifies the statement above.

Math 211 Linear Algebra

Addition, Scalar Multiplication, and Multiplication