CHAPTER 6

Linear Systems: Iterative Methods

In Section 4.1.1 and Section 4.1.4 we have discussed methods that obtain exact solution
of a linear system Ax = b in the absence of floating point errors (i.e., when the infi-
nite precision arithmetic is used). Such methods are called the direct methods. The
solution of a linear system can also be obtained using iterative procedures. Such meth-
ods are called iterative methods. There are many iterative procedures out of which
Jacobi and Gauss-Seidel methods are the simplest ones. In this chapter we introduce
these two basic iterative methods and discuss a sufficient condition under which these
methods converge. We introduce the notion of spectral radius of a matrix and prove a
necessary and sufficient condition on the spectral radius of the iterative matrix for the
convergence of a general iterative method to solve system of linear equations.

6.1 Jacobi Method

In Section 4.1.4, we have seen that when a linear system Ax = b is such that the
coefficient matrix A is a diagonal matrix, then this system can be solved very easily.
We explore this idea to build a new method based on iterative procedure. For this, we
first rewrite the matrix A as
A = D − C,
where D, C ∈ Mn (R) are such that
 
a11 0 ··· 0
 0 a22 ··· 0 
 
D= . .. .. ,
 .. . ··· . 
0 0 ··· ann

83
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.1 Jacobi Method

where aii , i = 1, 2, · · · , n are the diagonal elements of the matrix A. Then the given
system of linear equations can be rewritten as
Dx = Cx + b. (6.1)
If we assume that the right hand side is fully known to us, then the above system can
be solved very easily as D is a diagonal matrix. But the right hand side involves the
unknown vector x and therefore is unknown to us. Rather, if we choose (arbitrarily)
some specific value for x, say x = x(0) , on the right hand side then the resulting system
Dx = Cx(0) + b
can be readily solved for x on the left hand side. Let us call the solution of this system
as x(1) . That is,
Dx(1) = Cx(0) + b.
Now taking x = x(1) on the right hand side of (6.1) we can obtain the value of x on
the left hand side, which we denote as x(2) . Repeat this procedure to get a general
iterative procedure as
Dx(k+1) = Cx(k) + b, k = 0, 1, 2, · · · .
If D is invertible, then the above iterative procedure can be written as
x(k+1) = Bx(k) + c, k = 0, 1, 2, · · · , (6.2)
where B = D−1 C and c = D−1 b. The iterative procedure (6.2) is called the Jacobi
method and the matrix B is called the Jacobi iterative matrix.

Example 6.1.1.
Let us illustrate the Jacobi method in the case of 3 × 3 system

a11 x1 + a12 x2 + a13 x3 = b1

a21 x1 + a22 x2 + a23 x3 = b2
a31 x1 + a32 x2 + a33 x3 = b3 ,

where a11 $= 0, a22 $= 0, and a33 $= 0. We can rewrite the above system of linear
equations as
1
x1 = (b1 − a12 x2 − a13 x3 )
a11
1
x2 = (b2 − a21 x1 − a23 x3 )
a22
1
x3 = (b3 − a31 x1 − a32 x2 )
a33

84
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.1 Jacobi Method

Let x(0) = (x1 , x2 , x3 )T be an initial guess to the true solution x, which is chosen
(0) (0) (0)

arbitrarily. Define a sequence of iterates (for k = 0, 1, 2, · · · ) by


(k+1) 1 (k) (k) 
x1 = (b1 − a12 x2 − a13 x3 )  

a11 

1 
(k+1) (k) (k)
x2 = (b2 − a21 x1 − a23 x3 ) (JS)
a22 



1 
(b3 − a31 x1 − a32 x2 ). 
(k+1) (k) (k)
x3 = 
a33

which is the Jacobi iterative sequence given by (6.2) in the case of 3 × 3 system.
Now, the question is that will the sequence of vectors {x(k+1) } generated by the iterative
procedure (6.2) always converge to the exact solution x of the given linear system?
The following example gives a system for which the Jacobi iterative sequence converges
to the exact solution.
Example 6.1.2.
The Jacobi iterative sequence for the system
6x1 + x2 + 2x3 = −2,
x1 + 4x2 + 0.5x3 = 1,
−x1 + 0.5x2 − 4x3 = 0.

is given by 1
(k+1) (k) (k)
x1 = (−2 − x2 − 2x3 ),
6
(k+1) 1 (k) (k)
x2 = (1 − x1 − 0.5x3 ),
4
(k+1) 1 (k) (k)
x3 = − (0 + x1 − 0.5x2 ).
4
The exact solution (upto 6-digit rounding) of this system is
x ≈ (−0.441176, 0.341176, 0.152941).

Choosing the initial guess x(0) = (0, 0, 0), we get

x(1) ≈ (−0.333333, 0.250000, 0.000000),
x(2) ≈ (−0.375000, 0.333333, 0.114583),
x(3) ≈ (−0.427083, 0.329427, 0.135417),
x(4) ≈ (−0.433377, 0.339844, 0.147949),
x(5) ≈ (−0.439290, 0.339851, 0.150825),

85
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.1 Jacobi Method

and so on. We observe from the above computed results that the sequence {x(k) }
seems to be approaching the exact solution.

In the following example we discuss a system for which the Jacobi iterative sequence
does not converge to the exact solution.

Example 6.1.3.
Consider the system

x1 + 4x2 + 0.5x3 = 1,
6x1 + x2 + 2x3 = −2,
−x1 + 0.5x2 − 4x3 = 0.

which is exactly the same as the system discussed in Example 6.1.2 but the only
difference is the interchange of first and second equation. Hence, the exact solution
is same as given in (??). The Jacobi iterative sequence for this system is given by
(k+1) (k) (k)
x1 = (1 − 4x2 − 0.5x3 ),
(k+1) (k) (k)
x2 = (−2 − 6x1 − 2x3 ),
(k+1) 1 (k) (k)
x3 = − (0 + x1 − 0.5x2 ).
4
Choosing the initial guess x(0) = (0, 0, 0), we get

x(1) ≈ (1, −2, 0),

x(2) ≈ (9, −8, −0.5),
x(3) ≈ (33.25, −55, −3.25),
x(4) ≈ (222.625, −195, −15.1875),
x(5) ≈ (788.59375, −1307.375, −80.03125),

and so on. Here, we observe a diverging trend in the sequence {x(k) }.

The above two examples shows that the Jacobi sequence need not converge always and
so we need to look for a condition on the system for which the Jacobi iterative sequence
converges to the exact solution.

Define the error in the k th iterate x(k) compared to the exact solution by

e(k) = x − x(k) .

86
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.1 Jacobi Method

It follows easily that e(k) satisfies the system

e(k+1) = Be(k) ,

where B is as defined in (6.2). Using any vector norm and the matrix norm subordinate
to it in the above equation, we get

&e(k+1) & = &Be(k) &

≤ &B&&e(k) &
≤ ···
≤ ···
≤ &B&k+1 &e(0) &.

Thus, when &B& < 1, the iteration method (6.2) always converges for any initial guess
x(0) .
Again the question is
“what are all the matrices A for which the corresponding matrices B in (6.2) have the
property &B& < 1, for some matrix norm subordinate to some vector norm?”
We would like an answer that is ‘’easily verifiable” using the entries of the matrix
A. One such class of matrices are the diagonally dominant matrices, which we define
now.

Definition 6.1.4 [Diagonally Dominant Matrices].

A matrix A is said to be diagonally dominant if it satisfies the inequality
n
+
|aij | < |aii |, i = 1, 2, · · · , n.
j=1,j"=i

We now prove the sufficient condition for the convergence of the Jacobi method. This
theorem asserts that if A is a diagonally dominant matrix, then B in (6.2) of the Jacobi
method is such that &B&∞ < 1.

Theorem 6.1.5 [Convergence theorem for Jacobi method].

If the coefficient matrix A is diagonally dominant, then the Jacobi method (6.2)
converges to the exact solution of Ax = b.

87
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.1 Jacobi Method

Proof.
Since A is diagonally dominant, the diagonal entries are all non-zero and hence the
Jacobi iterating sequence x(k) given by
, n
-
1 +
(6.3)
(k+1) (k)
xi = bi − aij xj , i = 1, 2, · · · , n.
aii j=1,j"=i

is well-defined. Each component of the error satisfies

n
+ aij
(6.4)
(k+1) (k)
ei =− ej , i = 1, 2, · · · , n.
j=1
aii
j"=i

which gives
n .
+ .
. aij . (k)
. . &e &∞ . (6.5)
(k+1)
|ei | ≤ . aii .
j=1
j"=i

Define n . .
+ . aij .
µ = max . .
. aii . . (6.6)
1≤i≤n
j=1
j"=i

Then
(6.7)
(k+1)
|ei | ≤ µ&e(k) &∞ ,
which is true for all i = 1, 2, · · · , n. Therefore, we have

&e(k+1) &∞ ≤ µ&e(k) &∞ . (6.8)

The matrix A is diagonally dominant if and only if µ < 1. Then iterating the last
inequality we get
&e(k+1) &∞ ≤ µk+1 &e(0) &∞ . (6.9)
Therefore, if A is diagonally dominant, the Jacobi method converges.

Remark 6.1.6.
Observe that the system given in Example 6.1.2 is diagonally dominant, whereas
the system in Example 6.1.3 is not so.

88
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.2 Gauss-Seidel Method

6.2 Gauss-Seidel Method

Gauss-Seidel method is a modified version of the Jacobi method discussed in Section
6.1. We demonstrate the method in the case of a 3 × 3 system and the method for a
general n × n system can be obtained in a similar way.

Example 6.2.1.
Consider the 3 × 3 system

a11 x1 + a12 x2 + a13 x3 = b1 ,

a21 x1 + a22 x2 + a23 x3 = b2 ,
a31 x1 + a32 x2 + a33 x3 = b3 .

When the diagonal elements of this system are non-zero, we can rewrite the above
system as
1
x1 = (b1 − a12 x2 − a13 x3 ),
a11
1
x2 = (b2 − a21 x1 − a23 x3 ),
a22
1
x3 = (b3 − a31 x1 − a32 x2 ).
a33
Let
(0) (0) (0)
x(0) = (x1 , x2 , x3 )T
be an initial guess to the true solution x. Define a sequence of iterates (for k =
0, 1, 2, · · · ) by 
(k+1) 1 (k) (k) 

x1 = (b1 − a12 x2 − a13 x3 ), 

a11 

1 
(k+1) (k+1) (k)
x2 = (b2 − a21 x1 − a23 x3 ), (GSS)
a22 



1 
). 
(k+1) (k+1) (k+1)
x3 = (b3 − a31 x1 − a32 x2 
a33
This sequence of iterates is called the Gauss-Seidel iterative sequence and the
method is called Gauss-Seidel Iteration method.

Remark 6.2.2.
Compare (JS) and (GSS).

89
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.2 Gauss-Seidel Method

Theorem 6.2.3 [Convergence theorem for Gauss-Seidel method].

If the coefficient matrix is diagonally dominant, then the Gauss-Seidel method con-
verges to the exact solution of the system Ax = b.

Proof.
Since A is diagonally dominant, all the diagonal elements of A are non-zero, and
hence the Gauss-Seidel iterative sequence given by
/ i−1 n
0
1 + +
(6.10)
(k+1) (k+1) (k)
xi = bi − aij xj − aij xj , i = 1, 2, · · · , n.
aii j=1 j=i+1

is well-defined. The error in each component is given by

i−1
+ +n
aij aij (k)
(6.11)
(k+1) (k+1)
ei =− ej − e , i = 1, 2, · · · , n.
j=1
aii a j
j=i+1 ii

For i = 1, 2, · · · , n,, define

i−1 .
+
.
+n . .
. aij . . aij .
αi = . . . .,
. aii . , βi = . aii .
j=1 j=i+1

with the convention that α1 = βn = 0. Note that µ given in (6.6) can be written as

µ = max (αi + βi )
1≤i≤n

Since A is a diagonally dominant matrix, we have µ < 1. Now

(6.12)
(k+1)
|ei | ≤ αi &e(k+1) &∞ + βi &e(k) &∞ , i = 1, 2, · · · , n.

Let l be such that

(k+1)
&e(k+1) &∞ = |el |.
Then with i = l in (6.12),

&e(k+1) &∞ ≤ αl &e(k+1) &∞ + βl &e(k) &∞ . (6.13)

90
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.3 Stopping Criteria

Since µ < 1, we have αl < 1 and therefore the above inequality gives
βl
&e(k+1) &∞ ≤ &e(k) &∞ . (6.14)
1 − αl
Define
βi
η = max . (6.15)
1≤i≤n 1 − αi

Then the above inequality yields

&e(k+1) &∞ ≤ η&e(k) &∞ . (6.16)

Since for each i,

βi αi [1 − (αi + βi )] αi
(αi + βi ) − = ≥ [1 − µ] ≥ 0, (6.17)
1 − αi 1 − αi 1 − αi
we have
η ≤ µ < 1. (6.18)
Thus, when the coefficient matrix A is diagonally dominant, Gauss-Seidel method
converges.

Remark 6.2.4.
A careful observation of the proof of the above theorem reveals that the Gauss-
Seidel method converges faster than the Jacobi method by comparing (6.18) and
(6.9).

6.3 Stopping Criteria

Let x∗ denote the computed solution using some method. The mathematical error
in the approximate solution when compared to the exact solution of a linear system
Ax = b is given by

e = x − x∗ . (6.19)

Recall from Chapter 3 that the mathematical error is due to the approximation made
in the numerical method where the computation is done without any floating-point
approximation ( ie., without rounding or chopping). Observe that to get the mathe-
matical error, we need to know the exact solution. But an astonishing feature of linear
systems (which is not there in nonlinear equations) is that this error can be obtained

91
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.3 Stopping Criteria

exactly without the knowledge of the exact solution. To do this, we first define the
residual vector

r = b − Ax∗ (6.20)

in the approximation of b by Ax∗ . This vector is also referred to as residual error.

Since b = Ax, we have

r = b − Ax∗ = Ax − Ax∗ = A(x − x∗ ).

The above identity can be written as

Ae = r. (6.21)

This shows that the error e satisfies a linear system with the same coefficient matrix
A as in the original system Ax = b, but a different right hand side vector. Thus, by
having the approximate solution x∗ in hand, we can obtain the error e without knowing
the exact solution x of the system.
In the iterative methods discussed above, we have a sequence {x(k) } that is expected
to converge to the exact solution of the given linear system. In practical situation, we
cannot go on computing the x(k) indefinitely and we need to terminate our computation
once the value of x(k) reaches a desired accuracy for a sufficiently large k. That is, when
the error
&e(k) & = &x − x(k) &
in the k th iteration in some norm is sufficiently small. Since, we do not know the exact
solution x, the error given above cannot be computed easily and needs another linear
system (6.21) to be solved. Therefore, the question is how to decide where we have to
stop our computation (without solving this linear system)? In other words, how do we
know whether the computed vector x(k) at the k th iteration is sufficiently close to the
exact solution or not. This can be decided by looking at the residual error vector of
the k th iteration defined as

r (k) = b − Ax(k) . (6.22)

Thus, for a given sufficiently small positive number $, we stop the iteration if

&r (k) & < $,

for some vector norm & · &.

92
S. Baskar and S. Sivaji Ganesh Spring 2022-23
 Section 6.4 Exercises

6.4 Exercises
Iterative Methods
1. Let A be a diagonally dominant matrix such that aij = 0 for every i, j ∈
{1, 2, · · · , n} such that i > j + 1. Does naive Gaussian elimination method
preserve the diagonal dominance? Justify your answer.

2. Let A be a diagonally dominant matrix. Show that all the diagonal elements of
A are non-zero (i.e., aii $= 0 for i = 1, 2, · · · , n.). As a consequence, the iterating
sequences of Jacobi and Gauss-Seidel methods are well-defined if the coefficient
matrix A in the linear system Ax = b is a diagonally dominant matrix.

3. Write the formula for the Jacobi iterative sequence of the system

7x1 − 15x2 − 21x3 = 2,

7x1 − x2 − 5x3 = −3,
7x1 + 5x2 + x3 = 1.

Without performing the iterations, show that the sequence does not converge to
the exact solution of this system. Can you make a suitable interchange of rows
so that the resulting system is diagonally dominants?
4. Find the n×n matrix B and the n-dimensional vector c such that the Gauss-Seidel
method can be written in the form

x(k+1) = Bx(k) + c, k = 0, 1, 2, · · ·

Here B is called the iterative matrix for the Gauss-Seidel method.

93
S. Baskar and S. Sivaji Ganesh Spring 2022-23