Theoretical Numerical Analysis: An Introduction to Advanced Techniques

Index

PART I

BASIC CONCEPTS

Numerical analysis or, as it is perhaps more appropriately called, computational mathematics is the study of methods for the numerical (and, in general, approximate) solution of mathematically posed problems. Loosely speaking one can distinguish between two different aspects of computational mathematics: methodology, which deals with the construction of specific algorithms, their efficiency, implementation for computers, and various other practical questions, and analysis, in which one studies the underlying principles, error bounds, convergence theorems, and so on. It is this second aspect that is our main concern.

Most introductory treatments of numerical analysis stress methodology, and the student is exposed to a number of different algorithms which, on the surface, seem to have little in common. A deeper study, however, reveals a different picture. There are a few basic principles on which most of computational mathematics is based; often seemingly different methods are based on the same general idea. It is our aim to study these basic principles and to single out the fundamental notions which are the heart of modern numerical analysis. No attempt will be made to discuss particular problems or algorithms, except as examples for a general theory. To understand the principles on which numerical analysis is based is not only of interest to the mathematician, but can also be of considerable help in practical computation since a clear idea of the fundamentals is valuable in the construction of solution methods for the difficult problems encountered in practice.

We will begin by reviewing some simple results from functional analysis and approximation theory. A knowledge of these concepts is indispensable for further work; functional analysis provides us with a language for the development of a general theory, while approximation theory makes it possible to relate this theory to specific cases in a meaningful way. Both subjects are of course very extensive and contain topics of little use to the numerical analyst. Fortunately, the ideas needed here are the simpler ones, so that we need not delve into these subjects very deeply.

1

REVIEW OF FUNCTIONAL ANALYSIS

Modern numerical analysis has benefited considerably from its close association with functional analysis, which provides it not only with an elegant and concise notation, but also with an efficient tool for the development of new theories. Because of this, it is becoming increasingly more difficult for anyone unfamiliar with functional analysis to appreciate the current developments in numerical analysis. Fortunately, matters are not as difficult as they sometimes seem and it is not necessary to become an expert in functional analysis; what we mainly need here is a working knowledge of some quite simple concepts. This chapter is devoted to a review of the most basic ideas of functional analysis which, for the most part, are sufficient for further discussions. Occasionally, particularly in later chapters, a few more sophisticated concepts will be helpful, but these will be introduced when needed.

1.1 LINEAR SPACES

DEFINITION 1.1. Let X = {x,y,z...} be a set and F={α,β,γ,...} a scalar field. Let there be defined an operation of addition between every two elements of X, and an operation of scalar multiplication between every element of F and every element of X such that

(a) If x is in X and y is in X then x + y is in X . In mathematical shorthand notation we will write this as x ∈ X , y ∈ X ⇒ x + y ∈ X .

(b) x ∈ X , α ∈ F ⇒ αx ∈ X .

(c) x + y = y + x .

(d) ( x + y ) + z = x + ( y + z ).

(e) There exists an element 0∈ X such that x + 0 = x for all x ∈ X .*

(f) For each x ∈ X there exists a unique element, called the negative of x and denoted by — x , such that x + (– x ) = 0.

(g) α ( βx ) = ( αβ ) x .

(h) α ( x + y ) = αx + αy .

(i) ( α + β ) x = αx + βx .

(j) 1 x = x .

Then X is called a linear space over the field F. If F is the field of real numbers, X is a real linear space; if F is the field of complex numbers, X is a complex linear space. We normally deal only with real spaces and, unless otherwise stated, we use the term linear space to denote a real linear space.

Example 1.1. There are several familiar examples of linear spaces which play an important role in applied mathematics and numerical analysis:

(a) The set of all vectors with n components, with the usual definition of addition and scalar multiplication, is a linear space. We denote this space by R n . Much of the terminology for the general case has been adopted from this example. Thus a linear space is often called a vector space and its elements are referred to as vectors or points.

(b) The set of all continuous functions on some interval [ a,b ] is a linear space, denoted by C [ a,b ]. Again, addition and scalar multiplication are defined in the usual way, that is, pointwise.

(c) The set of all w-times continuously differentiable functions on [ a,b ] is a linear space, denoted by C ( n ) [ a , b ].

(d) The set of all functions x ( t ) for which

exists is the linear space Lp or Lp[R] The integral here is to be interpreted in the Lebesgue sense, but the reader who is unfamiliar with Lebesgue integration may, without much loss, think of it as a Riemann integral. Of particular interest in mathematics is L2, the space of all square-integrable functions.

(e) The set of all infinite sequences x 1 , x 1 ,... of real numbers such that

is a linear space, denoted by lp.

* The 0 here denotes the zero element in X and should, strictly speaking, be distinguished from the zero element of F. But since the context always removes any possible ambiguity we use the same symbol for both zeros.

DEFINITION 1.2. Let x1, x2, ... be elements of a linear space, and α1, α2,... be elements of the underlying field. The sum

is called a linear combination of the xi. The elements xi are said to be linearly independent if and only if

Otherwise, if there exists a set α1, α2,..., αn, not all zero, such that the linear combination is zero, the set is said to be linearly dependent.

DEFINITION 1.3. If there exist elements x1,x2,...,xn ∈ X which are linearly independent, but every set of n + 1 elements is linearly dependent, then we call n the dimension of X, denoted by dim(X). If for every n > 0 there exist n linearly independent elements in X, then X is said to be of infinite dimension.

DEFINITION 1.4. A linearly independent set x1,x2,..., xn∈X is said to be a basis for X if every x ∈ X can be expressed as a linear combination of the xi.

This definition of a basis is appropriate when X is of finite dimension; for infinite-dimensional spaces the analogous definition requires the notion of a convergent series and will have to be deferred until the next section.

THEOREM 1.1. A linear space X is of finite dimension n if and only if it has a basis of n elements. Also, any set of n linearly independent elements x1, x2,...,xn constitutes a basis for an n-dimensional space, called the span of x1,...,xn.

The proof is left as an exercise.

Example 1.2

(a) The set of all vectors with n components forms an n -dimensional space.

(b) The set of all polynomials of degree not greater than n of dimension n + 1. The powers 1, t , t ² ,..., t n .

(c) The space of continuous functions C [ a , b ] has infinite dimension.

Verification of these statements is left to the reader.

Exercises 1.1

1. If X is a linear space [ Definition 1.1 , (a)-(j)] show that for every x ∈ X

2. Show that the set of solutions of the differential equation

is a linear space. What is the dimension of this space?

3. Show that L x and l 1 are linear spaces.

4. Show that the functions 1, e t , e ² t , e ³ t are linearly independent over any interval [ a,b ].

5. Prove Theorem 1.1 .

7. Show that C [ a , b ] has infinite dimension.

8. What is the dimension of the space spanned by

9. Does the set of functions of the form x ( t )=1/( a + bt ) constitute a linear space?

10. Show that if { x n } ∈ l p , then { x n } ∈ l p , when p ′ > p > 0.

11. Give an example of a function which is in L 1 [0,1] but not in L 2 [0,1].

1.2 NORMS

When analyzing approximation methods we often need to compare solutions or to measure the difference between various answers. In the terminology of linear spaces we must find the distance between two points in the space. Thus we want to generalize the notions of distance and vector length. The introduction of a norm provides this generalization.

DEFINITION 1.5. With each x in a linear space X we associate a nonnegative number, called the norm of x and denoted by ||x||, such that

(a) || x || ≥ 0 for all x ∈ X .

(b) || x || = 0 if and only if x = 0.

(c) || αx || = | α ||| x || for all x ∈ X , α ∈ F .

(d) The triangle inequality

is satisfied for all elements x,y ∈ X.

Then X is said to be a normed linear space. The nonnegative number ||x − y|| is called the distance between the points x and y.

Example 1.3. In Rn we can define the norm of a vector

by

We call this the Euclidean norm and it is the usual geometric length of the vector. More generally, for any integer p ≥ 1 we can define a norm by

To show that this does in fact define a norm we must show that conditions (a)−(d) of Definition 1.5 are satisfied. The first three of these are trivial; the triangle inequality (d) is essentially the well-known Minkowski inequality (Davis, 1963, p. 132)

The space Rn with pis the familiar Euclidean n-space, but several other choices for p are of importance in numerical analysis. With p = 1 we have

Here the length of a vector is measured by the sum of the magnitudes of its components. Another possibility is to use the magnitude of the largest component as a measure of length. Then

This is the so-called maximum or infinity norm. The latter name, together with the notation, comes from the fact that

(See Exercise 3 at the end of this section.)

Example 1.4. Analogous definitions can be made for the space Lp. For x(t) ∈ Lp[a,b] we define the norm by

If x(t) is bounded in [a,b] the infinity norm is defined as

In the language of abstract mathematics, by introducing a norm we have introduced a topology into the linear space and we can now generalize the geometric concepts such as neighborhoods, convergence, and so on. There are, of course, many other ways to define a topology. We could, for instance, use the more general notion of a metric to define the distance between two points, and thus be led to consider metric spaces instead of the less general normed spaces (see Exercise 5). However, there is little to be gained for our purpose in such a generalization; most of the problems in numerical analysis can be discussed adequately in the setting of normed linear spaces which we have chosen here.

We can now extend the idea of a basis to infinite-dimensional spaces.

DEFINITION 1.6. A set of elements x1,x2,... of a normed linear space X is said to be closed (or complete) in X if every x ∈ X can be approximated arbitrarily closely (in norm) by a finite linear combination of the xi In other words, for any x ∈ X > 0 there exists an n and scalars α1,α2,..., αn such that

If the set {xi} is closed in X and linearly independent (that is, all finite subsets are linearly independent), then {xi} is called a basis for X and we write

Spaces which possess a countable (or finite) basis are said to be separable. Spaces which are not separable play no significant role in numerical analysis and are therefore of no interest to us here. We shall always assume, without further mention, that all our spaces are separable.

Exercises 1.2

1. Determine which of the following expressions are valid as definitions for norms in C ( n ) [ a , b ].

(a) max| x ( t )| + max| x ′( t )|.