Linear Algebra by Prof. R.

Vittal Rao

Lecture 2: August 10, 2010

an easy system for y. By analysing this system we get our analysis

for x.
2.1 Diagonalization Hence the question that arises is the following:
In the context of reducing a system of n equations in n unknowns
to an easy diagonal system, we have been looking at the question of Question 2.1 Given an n × n matrix A, can we find an invertible
diagonalization. Consider an n × n matrix and look at the system matrix P such that P−1 AP is a diagonal matrix?

Ax = b (2.1) We remark that the above question has been loosely formulated
and requires more precision, which we shall as see as we analyse
for b ∈ Fn (where F = RorC). further.
We introduce a change of variables,

y = Cx and c = Cb. Remark 2.1 Not all matrices are diagonalizable. Consider the ma-
trix !
where C is an n × n invertible matrix. Then the system (2.1) can be 0 1
A= .
written as 0 0
AC−1 y = C−1 c
We observe that
We denote by ! ! !
P=C −1 0 1 0 1 0 0
A =
2
= = O2 ,
0 0 0 0 0 0
Then the system (2.1) can be written as
where O2 denotes the 2 × 2 zero matrix.
P−1 APy = c (2.2) Suppose A is diagonalizable i.e., suppose ∃ invertible P such that
Suppose we can find P such that P−1 AP = D, a diagonal matrix, d1 0
!
then (2.2) becomes, P AP = D =
−1
; a diagonal matrix
0 d2
Dy = c,

1
2.1 Diagonalization
! 2.1.1 Some examples
d21 0
⇒ (P AP)(P AP) = D =
−1 −1 2
0 d22 !
d21
!
0 0 1
⇒ (P−1 AP)(P−1 AP) = D2 = Example 2.1 Consider the matrix, A =
! 0 d22 1 0
d21 0 We can think of A as a matrix over R.
⇒ P A P=
−1 2
0 ! d22 Now look at !the matrix,
d21 0 1 1
⇒ O2 = since A2 = O2 P=
0 d22 1 −1
!
⇒ d1 = d2 = 0 ⇒ D = O2 1 1
Then P = 1
−1 2 2
⇒ P−1 AP = O2 2
− 12
⇒ A = O2 Then P and P−1 are
! also matrices over R and we have
0 1
A contradiction. P−1 AP = , a real diagonal matrix.
1 0
Thus, in this example the real matrix A can be diagonalized over
R (and hence over C since P can be thought of as a matrix over C).
Thus there are matrices which cannot be diagonalized. Hence
the first basic question in the context of diagonalization is the
following: 0 −1
!
Example 2.2 Consider the matrix, A =
1 0
Suppose ∃P such that
Question 2.2 What is the criterion [C] that A should satisfy in !
d1 0
order that A is diagonalizable? P AP =
−1
0 d2
!
d21 0
⇒P A P=
−1 2
0 d22
Remark 2.2 Once again we remark that we shall fine tune this !
d21 0
question as we go along since the answer to the question also ⇒ −I = since A2 = −I
depends on where are looking for the P. We shall illustrate this by 0 d22
some examples. ⇒ d21 = d22 = −1

2
2.2 The Basic Questions

⇒ d1 , d2 are complex
Remark 2.3 Note that if such a P exists it is not unique.
⇒ D is complex For if ∃ such a P then we have
⇒ P cannot be real (since then D = P−1 AP will also be real)
P−1 AP = D
⇒ A cannot be diagonalized
! over R. !
1 i
1 1 − If α , 0 then if Q = αP we get
Now let P = Then P = 21 i 2
−1
i −i 2 2
We can verify that P−1 = Q−1 and P = α−1 Q
!
i 0 Hence,
P AP =
−1
0 −i D = P−1 AP = αQ−1 Aα−1 Q = Q−1 AQ
Hence, A is diagonalizable over C, whereas it was not diagonaliz- Thus Q is also one such diagonalizing matrix for any α , 0
able over R (even though A is a real matrix). Hence, if we have one P then there are infinite number of such P.
The idea will be to get a P gor which P−1 computation is as easy as
possible.
The above examples suggest that we must make it precise about
the type of matrix P we are seeking for diagonalization.
We shall for the moment look for diagonalization over R or C. Remark 2.4 What we mean by compromise on transformation is
We shall denote by F (either R or C) and make it more precise later. the following:
We had introduced the same change of variable transformation
2.2 The Basic Questions y = Cx
c = Cb
Once we are given an n × n matrix A over F, the first question
we ask is the following: for both the known and the unknown. Suppose we seek different
transforms,say,
y = Cx
,
Question 2.3 Does A satisfy the criterion [C] for diagonalization? c = C1 b
We may get an answer YES or NO.
where C and C1 are invertible n × n matrices, then

These lead us to a string of questions which we list in Figure 2.1. Ax = b

3
2.2 The Basic Questions
Does A satisfy criterion [C]?

YES NO

There exists P s.t. P−1 AP

There is no P s.t. P−1 AP is diagonal
is a diagonal matrix D

What is the struc-

What can we do?
ture of such a P?

How do we find such

Compromise!
a P? (See remark 2.3)

Compromise on what?

Compromise on Compromise on transfor-

Diagonalization mation (See remark 2.4)
What are the diago-
nal entries of D Are
they uniquely fixed?
Ask for P such that Ask for P and Q invert-
−1
P AP is nearly diagonal ible matrices (n × n)
such that Q−1 AP is
a diagonal matrix
One such answer is to
triangularize (SCHUR de- Singular Value Decom-
composition and Jordan position (SVD) is an
decomposition are some answer in this direction
answers in this direction)
Can be generalized to
rectangular matrices as
well. (See remark 2.5)

Figure 2.1: Series of questions from criterion [C] for A

4
2.3 A Third Important Problem

We then define their dot product or inner product as

becomes
Q−1 APy = c n
X
(x, y) := y x =
T
xj yj
(where Q = C−1
1
, P = C−1 )
j=1
Now if we can choose P and Q (i.e. C and C1 ) such that Q−1 AP is
diagonal, then we have an easy system for y. The dot or inner product gives rise to a number.
We now define the outer or tensor product as
 
Remark 2.5 If we had an m × n matrix then b will be m × 1. So we  y1 
 .  
 
see Q an m × m matrix and P and n × n matrix such that 
x ⊗ y := yx =  .  x1 . . . xn
T  
 . 
Q−1 AP  
yn
 
has its leading largest diagonal block as a diagonal matrix.  
= ai j ; where ai j = y j x j
1≤i≤n,1≤j≤n

Thus, the tensor product is an n × n matrix

2.3 A Third Important Problem  y1 x1 y1 x2 . . . y1 xn


 y x y x . . . y x



 2 1 2 2 2 n
x ⊗ y =  ..

We have so far looked at two important problems, (which are .
. . . .
.

 . . . . 
related), namely,

yn x1 yn x2 . . . yn xn
 
1. linear systems, and
Note that x ⊗ y is generated by the single row vector xT since each
2. diagonalization of a matrix. row is a multiple of xT and the components of y are the multiplying
factors (We can also think of the matrix x ⊗ y as generated by the
We shall now look at another important problem.
single column vector y since all the the other columns are multiples
Consider two n × 1 matrices, (say real)
of y, and the components of x are the multiplying factors).
If x and y are nonzero, then x ⊗ y is an n × n matrix generated by
   
x1   y1 
 .   .  a single row vector or a single column vector. We say x ⊗ y is an
   
x =  .  ; y =  .  n × n matrix of rank 1.
   
 . 
 
 . 
  If x(1) , x(2) , . . . , x(r) is a linearly independent set of n × 1 vectors,
xn
 
yn
  and y(1) , y(2) , . . . , y(r) is a linearly independent set of r vectors (n × 1

5
2.4 Exercises

vectors), then

r
2.4 Exercises
X
x(j) ⊗ y(j)  0 1 0 
 
j=1 1. Let A =  0 0 1 
 
0 0 0
 
is an n × n matrix of rank r.
 0 0 0 
 
Similarly if we choose above the x(j) as n × 1 and y( j) as m × 1,
(a) Verify that A3 =  0 0 0  = O3 , the 3 × 3 zero matrix.
 
then
0 0 0
 
Xr
x(j) ⊗ y(j) (b) Examine whether A can be diagonalized over
j=1 i) F2 , ii) Q, iii) R, iv) C
!
is an m × n matrix of rank r. 0 2
2. Let A = . Consider A as a matrix over F7 .
1 0
The question that arises is the following: !
3 1
Let P = (as a matrix over F7 )
1 2

Question 2.4 Given any m × n real matrix A can we express it as (a) Find P−1 over F7
!
s 3 0
X (b) Verify that P−1 AP =
x(j) ⊗ y(j) 0 4
j=1 (and hence A is diagonalizable over F7 ).

for suitable x(1) , x(2) , . . . , x(s) ∈ Rn and y(1) , y(2) , . . . , y(s) ∈ Rm ?

If so, what is the minimum number of terms required,
(i.e., what is the minimal value of s) and how do we find
x(1) , x(2) , . . . , x(s) , y(1) , y(2) , . . . , y(s) ?

This leads us to again to SVD of A but now as a decomposition

of sum of matrices.