pTER CONTENTS cHAl INTRODUCTION ving Linear Systems by Factoring Numerical] Methods o 4U-Decompositions 479 £2 The Power Method ae eA patel Search Engines 495 8.5 Sig nso” Of Procedures for Solving Linear Systems. 00 es Sirguar Value Decomposition 505 “© Pata Compression Using Singular Value Decomposition 512 Ina ce cated with “numerical methods” of linear algebra, an area of study hoe ‘chniques for solving large-scale linear systems and for finding ;pproximations of various kinds. It is not our objective to discuss algorithms and technical issues in fine detail, since there are many excellent books on the subject. Rather, we will be concerned with introducing some of the basic ideas and exploring important contemporary applications that rely heavily on numerical ideas—singular Value decomposition and data compression. A computing utility such as MATLAB, Mathematica, or Maple is recommended for Sections 9.2 to 9.6. LU-Decompositions Up to now, we have focused on two methods for solving linear systems, Gaussian climination (reduction to row echelon form) and Gauss—Jordan elimination (reduction to reduced row echelon form). While these methods are fine for the small-scale problems in this text, they are not suitable for large-scale problems in which computer roundoff error, memory usage, and speed are concerns. In this section we will discuss a method for solving a linear system of n equations in n unknowns that is based on factoring its product of lower and upper triangular matrices. This method, coefficient matrix into a » i the basis for many computer algorithms in common use. called “LU -decomposition,” ction is to show how to solvea linear system Ax = b of n equations st goal in this s ba oan by factoring the coefficient matrix A into a product A=LU a “Lis lower triangular and U is upper triangular. Once we understand how to do where 1 i iGjseuss how to obtain the factorization itself. this, we wil aie i ye have somehow obtained the factorization in (1), the linear system Assuming Paolved by te following procedure, called LU-decomposition. Ax= 479480 Chapter 9 Numerical Methods ‘The Method of LU-Decomposition Step 1. Rewrite the system Ax = bas LUX Q) Step 2, Define anew n x | matrix y by ux=y ° ‘Step 3. Use (3) to rewrite (2) as Ly = band solve this system for y. ‘Step 4. Substitute y in (3) and solve for x. ‘This procedure, which is illustrated in Figure 9.1.1, replaces the single linear system Ax = b by a pair of linear systems Ux=y Ly=b that must be solved in succession. However, since each of these systems has a triangular coefficient matrix, it generally turns out to involve no more computation to solve the two systems than to solve the original system directly. > Figure 9.1.1 > EXAMPLE 1 Solving Ax =b by LU-Decomposition Later in this section we will derive the factorization [fa 9-B i pis -3--8 of=|-3 1 offo1 3 4.9 2 4-3 aloo4 @) u Abo es L Use this result to solve the linear system 206 2APAD Pp 3-8 0] |x]=]2 49) 2tbeld) Ls. 2 0 M3 Py 3 1 ollo 1 3} fy $rird i 7110.05 4 U ) f sud nNhi “wding LU-Decompositions 9.1 LU-Decompositions 481. As specified in Step 2 above, let us define yy, y2, and ys by the equation ° 3.1] Px y SE 0.0 1) Lxs. ys. ©) u x=y which allows us to rewrite (5) as, 2 0 O77» 2 3 1 O|;y|=]2 4-3 73 bys. 3. ” L y=b or equivalently as 2x -3y + yn 4y1 — 3y. + Tys This system can be solved by a procedure that is similar to back substitution, except that wwe solve the equations from the top down instead of from the bottom up. This procedure, called forward substitution, yields wel ye (verify). As indicated in Step 4 above, we substitute these values into (6), which yields the linear system [2 a. xy 1 ;O 13 mi =|5 Lo 0 i [: [ m+3m+ = 1 ay + 3x3 = 5 xy =2 or, equivatentiy, Solving this system by back substitution yields m=2, ma-l 6 =2 (verify). <4 Example 1 makes it clear that after A is factored into lower and upper triangular matrices, the system Ax = b can be solved by one forward substitution and one back substitution. We will now show how to obtain such factorizations. We begin with some terminology. DEFINITION 1 A factorization ofa square matrix A as A = LU, where Lis a e tion angular and U is upper triangular is calledan LU-decomposition (or LU-factorizal of A.482 Chapter 9 Numerical Methods : ition. However, we will see that igi. U-decomposition. we will See tha fy row echelon form by Gaussian elimination yi, ‘ 2 ition, performing any row interchanges, then A eu ba ead _ aetna, usit .ce of row operat We form U using a sequencs i ot these operations can be accomplished by muliphyigg Not every square matrix has a possible to reduce a square matrix ‘Aon the left by an appropriate sequence of elementary matrices; that is, there ex, elementary matrices Ey, E2,.-++ Ex such that Eye ExEA =U 6 Since elementary matrices are invertible, we can solve (8) for A as A= Eley! EU or more briefly as ® where Ls Ey'By' Ej! (10) We now have all of the ingredients to prove the following result: THEOREM 9.1.1 If A is a square matrix that can be reduced to a row echelon form U by Gaussian elimination without row interchanges, then A can be factored as A= LU, where L is a lower triangular matrix, Proof Jet L and U be the matrices in Hone'as $19} and (8), respectively. The matrix U is upper triangular because it is a rew eclicion torn ofa square matrix (so all entries below its main diagonal are zero). To prove that J. is lower triangular, it suffices to prove that each factor on the right side of (10) is lower triangular, since Theorem 1.7.16 will then imply that L itself is lower triangular. Since row interchanges are excluded, each E; results either by adding a scalar multiple of one row of an identity matrix to a row below or by multiplying one row of an identity matrix by a nonzero scalar. In either case, the resulting matrix E; is lower triangular and hence so is E7' by Theorem 1.7.1d. This completes the proof. > EXAMPLE 2 AnLU-Decomposition Find an LU-decomposition of 26 2 A=|-3 -8 0 4 9 2 Solution To obtain an LU-decomposition, A form U using Gaussian elimination and then follows: = LU, we will reduce A to a row echelot calculate L from (10). ‘The steps are© 84, LU-Decomposit Reduction to Elementary Mat Row Ecketon Form Row Operation Corresponding 4 the Row Operatioy Inverse of the a Elementary Matrix Don 6, pe y.2. 3 -8 0 4 9 2. }xrow1 700 ‘iV ols oo Zi=10 1 0 o1 | iz Oil 001 1 3 1 =3|>-8 0 44:97 £2. [1 0 o loo step 2 Gxrowl)+row2 E)=|3 1 9 Beles tito to 01 001 Tees 1] Orns ehyoci3 [agaivcg tun, f 100 100 Step 3 (-4 x row 1)+row3 E;=| 0 1 0} Bi'=]0 10 ; L401 L401 Teees ela Osh ative LO i-3 oi-Bd fl 0 0 1810)" 0 g) =| Oeil 1:0 3 x row 2)+row3 Ey 10) & a ps 031 o 3 1 fiers 1 Orga tee 3 O:Fat0- 71 fi 0 0 10 | ol =|0 10 Step 5 }xrow3 Es lo ay Aa Fie git-wes A) 0 1. 3\/eu lo o af484 Chapter 9 Numerical Methods Bookkeeping and, from (10), Po OTife aed 10] [i007] | HOH 0] Pik 0.0 celoracollcs. a0 |fot0] 10 ctige:0]] Osti00 oo 1{{ oo afl4 0 tLe -3 Lo 0 7 2 (in 01910) =|=3 1) :0 4 3007 fn 2:6 6 br? 2 0 ovf1 31 2328 Of |23 1° OF [O}1 3 gig *> 4-3 7J[0 01 is an LU-decomposition of A. “4 AsExample2 shows, most of the work in constructing an LU-decomposition is expen in calculating L. However, allthis work can be eliminated by some careful bookkeeping of the operations used to reduce A to U. Because we are assuming that no row interchanges are required to reduce A to U, there are only two types of operations involved—multiplying a row by anonzero constant and adding a scalar multiple of one row to another. The first operation is used to introduce the leading 1’s and the second to introduce zeros below the leading 1’s In Example 2, a multiplier of | was needed in Step 1 to introduce a leading 1 in the first row, and a multiplier of } was needed in Step 5 to introduce a leading 1 in the third row. No actual multiplier was required to introduce a leading 1 in the second row because it was already a 1 at the end of Step 2, but for convenience let us say that the multiplier was 1. Comparing these multipliers with the successive diagonal entries of L, we sce that these diagonal entries are precisely the reciprocals of the multipliers used to construct U 1 i L4 a a Also observe in Example 2 that to i we used the operations troduce zeros below the leading 1 in the first row, add 3 times the first row to the second add —4 times the first row to the third and to introduce the zero below the leading 1 in the second row, we used the operation add 3 times the second row to the third Now note in (12) that in each position below the main diagonal of L, the entry is the negative of the multiplier in the operation that introduced the zero in that position in U' i)9.1. LU-Decompositions “405 7 This suggests the following procedure for constructing an LU-decomposition of matrix A, assuming that this matrix ean be reduced to row echelon form wits interchanges. out row Procedure for Constructing an LU-Decomposition Step 1. Reduce A to a row echelon form U by Gaussian elimin: imterchanges, Keeping track of the multipliers used to introduce the leadine fy the multipliers used to introduce the zeros below the leading I's, cand ‘Step 2. In each position along the main diagonal of L, place th multiplier that introduced the leading 1 in that position in U. Step 3._ In each position below the main diagonal of L, place the negative ofthe multiplier used to introduce the zero in that position in U. ‘Step 4. Form the decomposition A = LU. ation without row reciprocal of the > EXAMPLE 3 Constructing an LU-Decomposition Find an LU-decomposition of 6 -2 07 A=|9 -1 1 Belin 3 Solution We will reduce A to a row echelon form U and at each step we will fill in an entry of L in accordance with the four-step procedure above. 6 -2 " fe 0 0 : = denotsan known A=|9 -1 1] oo 0} [earacs Sei detieks oe multiplier = + rela! as wo Sg . 729 Les <— multiplier = —9 <— multiplier = -3 — ©e- won nme u Gea seo eos £29 <— multiplier = 4 92 ou eS ° <— multiplier = -8 38 "No actual operation if aed eset Mere any wel toe a ° s cooreooroo- ee co“ Ss Bea no eco es megs e295 £25 oe Nie 6 " x ° <— multiplier = 1 [3 8486 Chapter 9 Numerical Methods LU-Decompositions Are Not Unique LDU-Decompositions ition “Thus, we have constructed the LU-decomPes! i 6 0 0 oe 0 2 aLu=|9 2 ‘ [; silo o 1 We leave it for you to confirm this end result by multiplying the factors, fe leave it for In the absence of restrictions, Lu-decompositions are not unique, For example ig in the absen . my 0 Off we ae AzLu=|hr '2 OL is Biriitsa ibs']' 0, OT and 1 has nonzero diagonal entries, then we can shift the diagonal entries from he yg factor to the right factor by writing 1 0 A=|in/it In/ln boa/ te 2, iy 0 OPP] m2 ts 0 In 0 0 1 uy 0 0 bs, o 0 1 =o 1 0 O07 fia Fume fiers fa/tn 1 Of | 0 ba taateas Iy/ln bafta 1S Lo 0 bs which is another LU-decomposition of A. ‘The method we have described for computing £4'-decompositions may result in an “asymmetry” in that the matrix U has 1's op th «:2'n diagonal but L need not. Howes if it is preferred to have 1’s on the main di > the lower triangular factor, then we can “shift” the diagonal entries of L to a dives! swirix D and write L as Lok where L’ is a lower triangular matrix with 1's on the main diagonal. For example? general 3 x 3 lower triangular matrix with nonzero entries on the main diagonal can’ factored as ans Ohi. 1 0 O}far 0 0 a an 0 | = | an/an 1 O} | 0 an 0 aan yt Las/an anfax 1J 0 0 ay, L L D Note that the columns of L’ are obtained by dividing each entry in the corresponsit a of L by the diagonal entry in the column, Thus, for example, we can rm um , 3 1 0 0 oO) 72 0 tooffet 3 1}l0 0piu-Decompositions One can prove that if A is a square matrix that can be red r luced to row echelon fas without row interchanges, then A can be factored uniquely ag stele A=LDU where L is a lower triangular matrix with 1’s on the ‘matrix, and U is an upper triangular matrix with 1’s on the LDU-decomposition (or LDU-factorization) of A. main diagonal, D is a dia, , gonal the main diagonal. This ig called Many computer algorithms for solving linear systems perform row interchanges to re- duce roundoff error, in which case the existence of an LU-decomposition is xr guar- anteed. However, it is possible to work around this problem by “preprocessing” the coefficient matrix A so that the row interchanges are performed prior to computing the LU-decomposition itself. More specifically, the idea is to create amatix 0 (called pare mutation matrix) by multiplying, in sequence, those elementary matrices that profes the row interchanges and then execute them by computing the product QA. This prodicg can then be reduced to row echelon form without row interchanges, so itis assured ra have an LU-decomposition QA =LU (13) the systems But it follows from (13) that Qb and hence can be solved using LU- Because the matrix Q is invertible (being a product of elementary matrices), Ax =b and QAx = Qb will have the same solutions. the latter system can be rewritten as LUx = decomposition. It is common to see Equation (13) expressed as A= PLU (4) in whict is called a PLU-decomposition or (PLU-factorization) of A. $$. oncept Review W-decomposition ils perenne Whether a square matrix has an U-decomposition, Findan LU-decor Usethe method mposition of a square matrix. + LDU-decomposition + PLU-decomposition + Find the LDU-decomposition of a square matrix. + Find a PLU-decomposition of a square matrix. of LU-decomposition to solve linear stems, False Exercises sular matrices, iy ul®-() determine whether the statement is true or (6) If Ly, Lay -.se Laren xn lower angular ; * and ju your answer. then the product Ly L +++ Le is lo. » ery squan ) Ita. square m, gular mat re matrix has an LU-decomposition. ition, then (d) If a square matrix A has an LU-decomposition, , DU-decomposition. ‘atrix A is row equivalent to an upper tri- A has a unique LI trix U, then A has an LU-decomposition. si (e) Every square matrix has a PLU-decompo: