MA423 Matrix Computations

Lecture 11: Perturbation Theory for Linear Systems

Rafikul Alam
Department of Mathematics
IIT Guwahati

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Outline

• Condition numbers
• Perturbation and sensitivity analysis of linear systems

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Condition number
Definition: Let A be an n × n nonsingular matrix. Then cond(A) := kAk kA−1 k
is called the condition number of A. If cond(A) is NOT too large then A is said
to be well-conditioned. If cond(A) is large then A is said to be ill-conditioned.

Note that for a subordinate matrix norm, we have cond(A) = kAk kA−1 k ≥ 1.

Remark: The determinant det(A) is not a good measure of ill-conditioning of A.

A := 10−1 In =⇒ det(A) = 10−n and cond(A) = 1.

 
1 1010
B := =⇒ det(B) = 1 and cond∞ (B) = (1 + 1010 )2 ' 1020 .
0 1
Notice that columns A are orthogonal whereas columns of B are √ nearly linearly
dependent. Indeed, cos θ = hBe1 , Be2 i/kBe1 k2 kBe2 k2 = 1010 / 1 + 1020 ' 1.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Condition number and non-singularity
If A is nonsingular then when is A + ∆A nonsingular?
k∆Ak
Fact: If k∆Ak kA−1 k < 1 or equivalently, cond(A) < 1, then A + ∆A is
kAk
nonsingular.

Proof: If possible, suppose that A + ∆A is singular. Then there is a vector x

such that kxk = 1 and (A + ∆A)x = 0.

Then x = −A−1 ∆Ax =⇒ 1 = kA−1 ∆Axk ≤ kA−1 k k∆Ak, which is a

contradiction. 

Remark: There is a ∆A such that k∆Ak kA−1 k = 1 and A + ∆A is singular. In

1
other words, the distance to nearest singular matrix ∝ .
cond(A)

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Sensitivity analysis of linear systems
Consider the linear system
    
1 12 ··· 1
n x1 b1
1 1
··· 1  x2  b2 
2 3 n+1
.  =  . .

 .. .. ..  
. . ··· .   ..   .. 
1 1 1
n n+1
··· 2n−1
xn bn
| {z }
Hilbert matrix H

The matrix H is known as a Hilbert matrix, and it is known to be notoriously

ill-conditioned.
 >
To see what this means, set x := 1 · · · 1 ∈ Rn and define b := Hx. Then x
is the solution of Hx = b.
Now we use matlab to solve the linear system and compare the computed
solution with the known solution x.
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Sensitivity of solutions of Hilbert systems
>> xx = hilb(12)\b; Warning: Matrix is close to singular or
badly scaled. Results may be inaccurate. RCOND = 2.602837e-17.
n kx − xxk∞ cond(H)
4 .4130030e-12 2.837500e+04
6 .6964739e-09 2.907028e+07
8 .7311487e-07 3.387279e+10
10 .2047785e-03 3.535233e+13
12 .2476695e-00 3.841961e+16
This would appear to justify the predictions that as n increases, roundoff errors
would accumulate and destroy all accuracy in the computed solution of a linear
system!
The Hilbert matrix is SPD but the computed solutions differ drastically from true
solutions. Is it the fault of the algorithm?
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Perturbation of linear system-I
Theorem: Let A be nonsingular and cond(A) := kAk kA−1 k. Consider the linear
systems Ax = b and Ax̂ = b + ∆b. Then
kx − x̂k k∆bk
≤ cond(A) .
kxk kbk
Moreover, the upper bound is attained for some ∆b.

Proof: We have x̂ − x = A−1 ∆b =⇒ kx − x̂k ≤ kA−1 k k∆bk. Now

Ax = b =⇒ kbk ≤ kAk kxk =⇒ 1/kxk ≤ kAk/kbk, which yields the bound. 

Residual bound: Let x̂ = ALG(A, b). Then the residual r := b − Ax̂ yields
Ax̂ = b − r = b + ∆b, where ∆b := −r . Hence we have the residual bound
kx − x̂k kr k
≤ cond(A) .
kxk kbk

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Example    
1000 999 −1 −998 999
Consider A := . Then A = .
999 998 999 −1000

Thus kAk∞ = kAk1 = kA−1 k∞ = kA−1 k1 = 1999. Hence

cond∞ (A) = cond1 (A) = (1999)2 = 3.996 × 106 .
       
1 1999 −1 −1 1997
Observe that A = and A = .
1 1997 1 −1999
   
1999 −2 −1
Set b := and ∆b := 10 . Consider system Ax̂ = b + ∆b. Then
1997 1
   
−1 1 19.97
x̂ = x + A ∆b = + . This shows that
1 −19.99
−2
kx − x̂k∞ 2 10 k∆bk∞
= 19.99 = (1999) = cond∞ (A) .
kxk∞ 1999 kbk∞
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Perturbation of linear system-II
Consider the systems Ax = b and (A + ∆A)x̂ = b + ∆b. Suppose that A is
nonsingular and k∆Ak kA−1 k < 1. Then
 
kx − x̂k cond(A) k∆Ak k∆bk
≤ +
kxk 1 − k∆Ak
kAk
cond(A) kAk kbk
 
k∆Ak k∆bk
. cond(A) + .
kAk kbk
Proof: We have
x̂ − x = −A−1 (∆Ax̂ − ∆b) =⇒ kx̂ − xk ≤ kA−1 k(k∆Ak kx̂k + k∆bk). Now
kx̂k ≤ kx̂ − xk + kxk =⇒ (1 − kA−1 k ∆Ak)kx − x̂k ≤ kA−1 k(k∆Ak kxk + k∆bk).
Now diving both sides by kxk and using the fact that
b = Ax =⇒ kbk ≤ kAk kxk =⇒ kbk/kxk ≤ kAk, we obtain the bound. 
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Example
 
1 1+δ
Consider A := , where δ > 0. Then
1−δ 1
(2 + δ)2
 
−1 1 1 −1 − δ
A = 2 . Hence cond∞ (A) = .
δ −1 + δ 1 δ2

For δ := 10−2 , we have cond∞ (A) = (201)2 = 40401.

    
1 1.01 x1 2.01
Consider the linear systems = whose solution is
0.99 1 x2 1.99
   
 > 1 1.01 2
x = 1 1 and x̂ = .
1 1 2
> >
Then x̂ = 2 0 . Note that ∆A = 10−2 e2 e1> and ∆b = 10−2 1 1 . We
 
have kx − x̂k∞ /kxk∞ = 1.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Condition number of Hilbert matrix

Consider the n × n Hilbert matrix Hn = hilb(n), where Hn (i, j) := 1/(i + j − 1)

√ 4n !
(1 + 2)
Theoretically, cond2 (Hn ) ≈ √ .
n
But cond2 (Hn ) computed by matlab reaches the maximum when n = 13 and
does not continue to grow when n > 13.
This happens due to finite precision arithmetic. It is known that
σmax (Hn ) := kHn k2 → π and σmin (Hn ) := 1/kHn−1 k2 → 0 as n → ∞.Hence
σmax (Hn ) π π
cond(Hn ) = ≈ ≈ .
σmin (Hn ) σmin (Hn ) + eps eps
Now π/eps = 1.4148e + 16 and cond2 (H13 ) = 4.7864e + 17.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Growth of condition number of Hilbert matrix

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC