NUMERICAL LINEAR ALGEBRA

(part b)
– System Condition
– Special Matrices
– Gauss-Seidel
– Non-linear Systems
 System Condition
 Condition number indicates ill-conditioning of a system.
 We estimate condition number using matrix norms.
Matrix norms:
 A norm is a measure of the size of a multi-component entity
(e.g., a vector)
x3
 n

x 1   x i  1-norm
 i 1 
x 2
1/ 2
 n 2
x 2  x e    xi 
2-norm
(Euclidean norm)
 i 1 
x1
1/ p
 n
p
x   x i  p-norm
p
 i 1  x2
 We can extend Euclidean norm for matrices:
1/ 2
 n n
2 
A e    a i , j  (Frobenius norm)
 i 1 j 1 
 There are other norms too…, e.g.,
 n 
A  max   aij 
 (row-sum norm)
1i  n
 j 1 

 n 
A  max   aij  (column-sum norm)
1 j  n
 i 1 

 Each of these norms returns a single (positive) value for the

characteristics of the matrix.
Matrix Condition Number:
 Matrix condition number can be defined as

Cond A  A  A1 ( Cond A  1 )

 If Cond [A] >> 1  ill-conditioned matrix

 It can be shown that
i.e., the relative error of the
computed solution cannot
x A
 Cond A be larger than the relative
x A error of the coefficients of
[A] multiplied by the
condition number.
For example;
[A]  contains element of t S.F.
(x)  will contain
(precision of 10-t)
elements of (t-c) S.F.
(precision of 10c-t)
Cond [A]  10c
EX 10.4 : Estimate the condition number of the 3x3 Hilbert matrix using row sum
norm
 1 1 / 2 1 / 3 Hilbert matrix is
A  1 / 2 1 / 3 1 / 4 inherently ill-
conditioned.
1 / 3 1 / 4 1 / 5

First normalize the matrix by dividing each row by the largest coefficient:

1 1 / 2 1 / 3 
A  1 2 / 3 1 / 2 
1 3 / 4 3 / 5

Row-sum norm:

1  1 / 2  1 / 3  1.833
1 1 / 2 1 / 3 
A  1 2 / 3 1 / 2  1  2 / 3  1 / 2  2.1667 A   2.35
1 3 / 4 3 / 5 1  3 / 4  3 / 5  2.35
Inverse of the scaled matrix:

 9  18 10  this part takes

A1   36 96  60

the longest time
of computation.
 30  90 60 

Row-sum norm:
 9  18 10 
A1   36 96  60 A1

  36  96   60  192
 30  90 60 

Condition number:

Cond A  (2.35)(192)  451.2  matrix is ill-conditioned.

e.g., for a single precision (7.2 digits) computation;

c  log( 451.2)  2.65 (7.2-2.65)=4.55 ~ 4 S.F. in the solution!

(precision of ~10-4)
 Special Matrices
 In engineering applications, special matrices are very common.
> Banded matrices

aij  0 if i  j  ( BW  1) / 2
BW
BW=3  tridiagonal system

> Symmetric matrices

aij  a ji or A  AT

> Sparse matrices (most elements are zero)

only black areas

are non-zero
 Application of elimination methods to sparse matrices are not
efficient (e.g., need to deal with many zeros unnecessarily).

 We employ special methods in working with these systems.

Cholesky Decomposition:
 This method is applicable to symmetric matrices.

 A symmetric matrix can be decomposed as

i 1
aki   lijlkj
lki 
j 1
for i  1,2,..., k  1
A  LL T
or lii
k 1
lkk  akk   lkj2
j 1

Symmetric matrices are very common in engineering applications. So, this

method has wide applications.
EX 11.2 : Apply Cholesky decomposition to

 6 15 55 
A  15 55 225
55 225 979
Apply the recursion relation:
k=1
l11  a11  6  2.4495
(k=2,i=1)
a21 15
l21    6.1237 l  a  l 2
 55  ( 6 .1237 ) 2
 4.1833
l11 2.4495 22 22 21

(k=3, i=1) (k=3, i=2)

a 55 a l l 225  6.1237(22.454)
l31  31   22.454 l32  32 21 31   20.916
l11 2.4495 l22 4.1833

l33  a33  l31

2
 l32
2
2.4495 
 979  (22.454) 2  (20.916) 2 L  6.1237 4.1833 

 6.1106 22.454 20.916 6.1106 
 Gauss-Seidel
 Iterative methods are strong alternatives to elimination methods

 In iterative methods solution is constantly improved so there is no

concern of round-off errors.

 As we did in root finding,

> start with an initial guess.
> iterate for refined estimates of the solution.

 Gauss-Seidel is one of the most commonly used iterative method.

 For the solution of

A( x)  (b)
Write each unknown in the diagonal in terms of the other
unknowns:
For a 3x3 system:
b1  a12 x2  a13 x3
x1  Strategy:
a11 - Choose initial guesses for x2 and x3
b2  a21 x1  a23 x3 - Calculate x1
x2 
a11 - Calculate new x2 using last x1 and x2
- calculate new x3 using last x1 and last x2
b3  a31 x1  a32 x2
x3  - iterate solution to sufficient accuracy
a33

 In Gauss Seidel, new estimates are immediately used in

subsequent calculations.

 Alternatively, old values (x1 , x2 , x3) are collectively used to

calculate new values (x1 , x2 , x3)  Jacobi iteration (not
commonly used)
Convergence in Gauss-Seidel:
 Gauss-Seidel is similar to the fixed-point iteration method in root
finding methods. As in the fixed-point iteration, Gauss-Seidel
also is prone to
> possibility of divergence
> possibility of slow convergence

 Convergence of the G-S method can be checked by the following

criteria:
i.e., absolute value of the diagonal
n coefficient in each of the row must be
aii   aij larger than sum of the absolute values of
j 1
j i all other coefficients in the same row.
(diagonally dominant system)

 Fortunately, many engineering applications fulfill this requirement.

EX 11.2 : Use Gauss-Seidel method to obtain the solution of
true results
3 x1  0.1x2  0.2 x3  7.85
x1 = 3
0.1x1  7 x2  0.3 x3  19.3 x2 = - 2.5
0.3 x1  0.2 x2  10 x3  71.4 x3 = 7

Gauss-Seidel iteration:
7.85  0.1x2  0.2 x3  19.3  0.1x1  0.3 x3 71.4  0.3 x1  0.2 x2
x1  x2  x3 
3 7 10
Assume initial guesses all 0.
7.85  0  0  19.3  0.1(2.616667)  0
x1   2.616667 x2   2.794524
3 7

71.4  0.3(2.616667)  0.2(2.794524)

x3   7.005610
10
For the second iteration, we repeat the process:

7.85  0.1(2.794524)  0.2(7.005610)  t  0.31%

x1   2.990557
3
 19.3  0.1(2.990557)  0.3(7.005610)
x2   2.499625  t  0.015%
7

71.4  0.3(2.990557)  0.2(2.499625)  t  0.0042%

x3   7.000291
10

The solution is rapidly converging to the true solution!

Improvement of convergence by relaxation:
 After each value of x is computed using Gauss-Seidel equations,
the value is modified by a weighted average of the old and new
values.

xinew  xinew  (1   ) xiold 0 2

If 0<<1  underrelaxation (to make a system converge)

If >1  overrelaxation (to accelerate the convergence )

 The choice of  is empirical and depends on the problem.

 Non-linear Systems
 In root finding methods, we developed a method up to a system of
two unknowns (two-equation Newton-Raphson method).
 Consider a system of n non-linear equations with n unknowns:
f1 ( x1 , x2 ,..., xn )  0
f 2 ( x1 , x2 ,..., xn )  0
...
f n ( x1 , x2 ,..., xn )  0

 We use iterative methods to solve non-linear systems.

 Two approaches:
> Successive substitution
> Multi-equation Newton-Raphson method.
Successive substitution:
 Similar to fixed-point iteration and Gauss-Seidel methods: solve
each non-linear equation for one unknowns. Iterate the solution
for new solutions. (hoping that the solution will converge).
 Shortcomings: 1. convergence depends on how the equations are
manipulated. 2. divergence if the initial guesses are not sufficiently
close.
 Better use the alternative method of N-R.

Newton-Raphson:
 We linearize the system using first-order Taylor series expansion.
For k-th row:
f k ,i f k ,i f k ,i
f k ,i 1  f k ,i  ( x1,i 1  x1,i )  ( x2,i 1  x2,i )  ...  ( xn ,i 1  xn ,i )
x1 x2 xn

set to zero unknowns

 re-arranging the terms
f k ,i f k ,i f k ,i f k ,i f k ,i f k ,i
( x1,i 1 )  ( x2,i 1 )  ...  ( xn ,i 1 )   f k ,i  x1,i  x2 ,i  ...  xn ,i
x1 x2 xn x1 x2 xn
Define matrices
 f1,i f1,i f1,i 
 x ...
 1 x2 xn   x1,i 1   f1,i   x1,i 
 f 2,i f 2,i f 2,i  x  f  x 
Z    x1 x2
...
xn  X i 1    2,i 1  Fi    2,i   X i    2 ,i 
   ...   ...   ... 
 ... ... ... ...       
f n ,i f n ,i  x
 n ,i 1   f n ,i   xn ,i 
 f n ,i ...
 x1 x2 xn 

Then,
Z i X i 1   Fi   Z i X i 
Last equation is in the form of Ax=b, and can be solved using
elimination methods (i.e., LU decomposition).