All nature is but art unknown to thee, All chance, direction which thou canst not see; Alexander

Pope, 1688-1744, An Essay on Man

Chapter 7

Interlude: Modeling Randomness

This chapter develops methods for computer simulations of random models. Simulations can build intuition into the "workings" of randomness and develop insight into how random processes differ from deterministic ones; the algorithmic approach one must take to write a program leads to a different understanding of the model complementary to theoretical derivations. In addition, computer simulations are sometimes the only method for grinding out numbers from equations that are difficult to solve. A generalized algorithmic language is used; it is in fact a subset of Pascal, since Pascal is an excellent language in which to express the flow of logic in an algorithm. The terminology is explained in Appendix 1; you should be able to understand it readily if you know any computer languagenot necessarily Pascal.

7.1 RANDOM NUMBER GENERATORS

At the heart of simulations of random models is a method for producing random numbersa procedure or function that will churn out number after number uniformly distributed in the interval [O, 1]. The method explained in this section is the method used by most programming languages which have built in random number generators. Actually, the random number generator will be a specific formula that produces random numbers in a completely deterministic way. This is a contradiction
183

to the very idea of randomness. Consequently, the numbers produced by the random number generator are often called pseudorandom because, although they have a very definite pattern, they appear to have no discernible pattern detectable without knowing the exact formula used. The fact that the same sequence of pseudorandom numbers is generated each time the generator is used is even useful in helping debug programs and understanding the results of the simulation. Randomness means "no pattern"; pseudorandomness means "no apparent pattern." Start with positive integers MULT (for multiplier), ADDR (for adder), and NORM (for normalizer). SEED is to be a pseudorandom number satisfying
O =s SEED < NORM

Each time a new random number is needed, it is produced from the previous valu of SEED by the formula
SEED := (MULT*SEED + ADDR) mod NORM

That is, first SEED gets multiplied by MULT, then ADDR is added on, and finally the remainder upon divisin by NORM is the new valu of SEED.
Example 1

Use these vales

MULT:=6 ADDR : = 5 NORM: =11

What vales of SEED will be produced if the initial valu of SEED is O? If the initial valu is 4?
Solution Each time the new valu of SEED is SEED := (6*SEED + 5) mod 11

With SEED initially O this sequence is generated: 0 5 2 6 8 9 4 7 3 1 0 5 2 6 8 . . . With SEED initially 4 this sequence is generated:

4 7 3 1 0 5 2 6 8 9 4 7 3 1 0 5 2 . . .
From Example 1 two facts are apparent: First, since the next number in the sequence is generated from only the valu of the previous number, if any number is generated again, the entire list is a repetition from that point on. Second, the "cycle length"the number of distinct numbers before repetition occurscan be ai most of length equal to the valu of NORM. This is so since the mod function produces the remainder upon divisin by NORM, which necessarily is a number between O and NORM-1; thus there are NORM many possible remainders NORM many possible vales of the random number SEED, that is; and as soon as one is repeated, repetition of the entire list occurs. A good random number
184

Interlude: Modeling Randomness

Chap. 7

generator would use vales of MULT, ADDR, and NORM so that the cycle length is large.
Example 2

Consider the generator

SEED := (6*SEED + 3) mod 7

Then with different initial vales of SEED, these sequences are generated: 0 1 4 5 3 2 6 5 0 1 4 5 3 0 3 0 3 2 1 2 1 2 1 6 4 6 4 6 4 5 5 5 5 5 5 0 2 6 5 3 . . . ... . . . . . .

Although the valu of NORM = 7 suggests that the cycle length might be the mximum valu of 7, the actual cycle lengths are small; if SEED is initially 5, then the random number generator is quite useless. The theory of what vales of the parameters result in good random number generators is complicated and more a subject of abstract algebra than of probability. Our interest here is in using a random number generator and the following theorem is stated without proof. *
Theorem

The random number generator will produce the mximum cycle length NORM of pseudorandom numbers with any initial valu of SEED under either of these conditions: 1. NORM is a power of 10. ADDR ends in (units digit) 1, 3, 7, or 9. MULT-1 is a mltiple of 20. 2. NORM is a power of 2. ADDR is odd. MULT-1 is a mltiple of 4.

Example 3

With NORM = 10, ADDR = 27, and MULT = 21, the mximum cycle length of NORM = 10 is obtained. To check this, we use
SEED := (2USEED + 27) mod 10

*See Donald E. Knuth, The Art of Computer Programming, Vol. 2 (Reading, Mass.: AddisonWesley Publishing Co. Inc., 1971), Chapter 3. Sec. 7.1 Random Number Generators
185

to produce each successive valu of SEED. With SEED initially O this sequence is obtained: 0 7 4 1 5 2 9 6 3 0 which has cycle length 10. With NORM = 16, ADDR = 7, and MULT = 21, the mximum c; length of NORM = 16 is obtained. Here
SEED := (21*SEED + 7) mod 16

yields this sequence with SEED initially 0:

O 7 10 9 4 11 14 13 8 15 2 1 12 3 6 5 O

which does in fact have cycle length 16.

Finding remainders when dividing by powers of 10 can be done quickly t human beings. For example, 237 mod 100 = 37; 3451 mod 10 = 1; 9087 mo*J 1000 = 87. But computers (using binary arithmetic) can find remainders me quickly when dividing by powers of 2. Programs'will run faster, therefore. NORM is a power of 2. Besides long cycle lengths, a random number generator must satisfy this condition: The result of any arithmetic computation in calculating SEED from the previous valu of SEED must not exceed the mximum integer that can be stor on the computer; otherwise, an overflow condition will result. For computers that store integers in 32 bits, for example, this mximum integer is 2,147,483,647 = 2 1. The mximum valu used in computing the next valu of SEED is
MULT*(maxmum valu of SEED) + ADDR

But the mximum valu of SEED is NORM - 1. Consequently, MULT, ADDR. and NORM should be chosen so that
MULT*(IMORM-1) + ADDR

does not exceed the mximum integer size. Example 4 Using the theorem, the following would produce random number generators wtfc mximum cycle length NORM:
MULT*(NORM-1) + ADDR mximum integer produced 63,500 26,012,256 1,003,620 7 2 3 , 2 7 4 , 4 7 6 Chap

NORM

ADDR

MULT

100 10,000 1024 32,768

186

41 4,857 57 8,485

641 2,601 981 22,073

Interlude: Modeling Randomness

In summary, since large cycle lengths are desirable, MULT, ADDR, and NORM should be chosen satisfying the conditions of the theorem in such a way that (NORM-1)*MULT + ADDR is cise to the mximum integer size. Then any initial valu of SEED will produce NORM many pseudorandom numbers without repetition. Note that there is no point in using an initial valu of SEED > NORM -1 since each succeeding valu of SEED will be less than NORM. Why? So far, the random number generator produces integers SEED with O =s SEED < NORM. To obtain a number in the unit interval, simply divide SEED by NORM. With an initial valu of SEED and constants MULT, ADDR, and NORM, pseudorandom numbers in the unit interval [O, 1) are obtained by these two instructions:
SEED := (MULT*SEED + ADDR) mod NORM; RND := SEED/NORM

Example 5 Write a program to genrate 100 random numbers in [O, 1], Solution
const NORM = 10000; ADDR = 4857; 2,601 MULT = 8601; <*~~ var I, SEED: nteger; RND: real; begin SEED:= 0; for l:= 1 to 100 do begin SEED:= (MULT.SEED + ADDR) mod NORM; RND:= SEED/NORM; write(RND:8:3); end; end.

Pseudorandom numbers produced in this way have a good chance of passing various tests for randomness. For example, the mean of the numbers is approximately 1/2; approximately one-tenth of them will fall into each of the intervals [O, .1), [.1, .2), . . ., [.9, 1). With any finite sequence, however, there can be no complete test of randomness. In addition, since the pseudorandom numbers are obtained by a deterministic formula, if the random model that is being simulated incorporates a "pattern" that coincides with the "pattern" of the random number generator formula, then the generator may prove to be a bad one for that model. Before using a particular random number generator (choice of NORM, ADDR, and MULT), one should perform a few simple checks. Have a listing of several
Sec. 7.1 Random Number Generators
187

hundred of them and visually inspect the list; do they appear to be random? Have the average printed out. Is it cise to 1/2? And so on. (See Problems 7.6 to 7.10.) Also note that a random number generator is only capable of producing NORM many distinct vales of RND. If more are required for a simulation, the vales of NORM, ADDR, and MULT should be changed. ExampJe 6: Simulating Bernoulli Triis Input a probability/ of success for each Bernoulli trial and an integer MAX; output MAX many Bernoulli trial simulations (MAX many letters "S" or "F")Soluion After generating each valu of RND as in Example 5, a decisin has to be made whether the valu of RND is to correspond to an S or to an F. But the probability that RND is less than p is exactly /, since the interval [O, p) occupies a fraction p of the entire unit interval [O, 1J. Thus RND simlales a success if RND </; otherwise, RND simulates a failure.
constNORM = 10000; ADDR = 4857; MULT = 8601 ;'"" "Zf(*o\ var I,SEED,MAX: nteger; RND,P:real; begin . ' .- ' SEED: =0; readln(P,MAX); for l;= 1,.tp MAX do f .'".",. begin SEED:= (MULT*SEED + ADDR) mod NORM; RND:= SEED/NORM; : if RND < P then write ('S') else write ('F'); .; end; '; end.

PROBLEMS
7.1. With the following vales of NORM, ADDR, MULT, and initial valu of SEED, write out the sequence of successive vales of SEED generated from SEED : = (MULT*SEED + ADDR) mod NORM. Start with SEED = 0.

H
(a) (b) (c) (d) (e) (fl
188

MULT

ADDR

NORM
7 16 16 10 10

3 3 3 7 9 4

9
Interlude: Modeling Randomness Chap. 7

For which of these is the cycle length equal to NORM? Which of them satisfy the conditions of the theorem? 7.2. If NORM = 1035, ADDR = 6355, and MULT = 2309, what is the mximum valu of MULT*SEED + ADDR obtained with an initial valu of SEED less than NORM? 7.3. Show that M mod NORM = M, mod NORM if and only if M - M, is a mltiple of NORM. Show that, consequently, if MULT - MULT, is a mltiple of NORM, then
(MULT*SEED + ADDR) mod NORM = (MULT,*SEED + ADDR) mod NORM

Henee MULT, ADDR, NORM produce the same random number generator as MULT,, ADDR, NORM. 7.4. Show that MULT, ADDR, NORM produce the same random number generator as MULT, ADDR,, NORM if ADDR - ADDRj is a mltiple of NORM.

Note that Problem 7.3 implies that there is no point in taking MULT larger than NORM. This is so since if MULT > NORM, then the same random number generator is obtained by subtracting NORM from MULT enough times so that the new valu of MULT < NORM. Similarly, Problem 7.4 shows that ADDR can be taken to be less than NORM without eliminating any generators.
7.5. What will happen if MULT = 1? Plot 10 successive vales of SEED if the initial valu of SEED is O and SEED: = (1*SEED + 17) mod 400. What is wrong with , this generator? In the- following program problems use the same vales of NORM, ADDR, and MULT. i 7.6. Find out what the mximum integer is on your computer. Then choose NORM, . ADDR, and MULT satisfying the theorem, but so that (NORM-1)*MULT + ADDR does not exceed this mximum. 7.7. Wih your choice of NORM, ADDR, and MULT, write a program that uses the v' initial valu of SEED equal to 0. Loop through until SEED returns to 0. Print out the number of cycles of the loop performed. This number should be NORM (if you have chosen your random number generator in accordance with the theorem). Also print out the average of all the RND vales. 7.8. Write a program that will input MAX equal to the number of random numbers RND \J : = SEED/NORM to be generated. Output all these numbers (make sure that MAX is not too large!) and check that they appear to be randomly distributed on [O, 1] without any obvious pattern. K. 7.9. With the same random number generator, write a program to input MAX. Compute MAX many vales of RND starting with the initial valu of SEED equal to 0. Print out the number of vales of RND in each of the intervals [O, .1), [.1, .2), . . ., [.9,1]. These should each be cise to MAX/10. Run the program with several inputs ofMAX. *j 7.10. With the same random number generator, write a program to loop through MAX distinct vales of RND (starting with initial valu of SEED equal to 0) and compute rthe number of times the current valu of RND is larger than the preceding valu of RND. This number should be approximately MAX/2.
Chap. 7 Problems

189

7.2 SIMULATING PROBABILITIES AND EXPECTATIONS

Recall that the frequency interpretation of probability claims that if a randa experiment is performed a large number MAX of times and the event A occur* < times, then P(A) = /MAX. To simlate P(A), input a large valu MAX, simlate the random experimeni MAX times, add 1 to SUM each time the simulation results in event A, and output SUM/MAX. Recall also that if X is a random variable, then E(X) is approximately average of the X vales over a large number of repetitions of the random exp iment. To simlate E(X), input a large valu MAX, simlate the random experimeni MAX times, add the valu of X on each to SUM, and output SUM/MAX. How large should MAX be so that the approximations are accurate? i IK answer requires the central limit theorem and the law of large numbers (Chapu 10). But consider this: If MAX = 1000 and SUM is an integer, then SUM/MA can only have at most three nonzero digits .to the right of the decimal point. 7 accuracy to this many places, MAX has to be taken considerably larger than l(i>X
Example 7

Three balls are tossed randomly into 5 cans. What is the probability that r land in different cans? Solution There are 53 ways to distribute all the balls into the cans, 5 which have the balls in different cans. Henee the exact probability is

5-4-3 = .48 5-5-5

To simlate the probabilities, input an integer MAX, then MAX many : simlate three container numbers CAN[1], CAN[2], CAN[3]; these are intesa from 1 to 5 representing the cans. If CAN[1], CAN[2], and CAN[3] are dista add 1 to SUM.
const NORM = 10000; ADOR = 4857; MULT = 8601 ; var l,SEED,MAX,SUIvl,J: nteger; RND: real; CAN: array (1..3] of nteger;

Interlude: Modeling Randomness

Chap.

begin SEED:= 0; SUM:= 0; readln(MAX);

for l:= 1 to MAX do begin for J:= 1 to 3 do begin SEED:= (MULT*SEED + ADOR) mod NORM; RND:= SEED/NORM; CAN[J]:= trunc(5*RND + 1); end; f (CAN[1J <>CAN[2])and (CAN[1J o CAN[3J) and (CAN[2] o CAN3J) then SUM:= SUM+1; end; writeln(SUM/MAX:10:4); end.

73 SIMULATING DISCRETEDISTRIBUTIONS WITH A FINITE NUMBER OF OUTCOMES Example 6 simulaed Bernoulli triis in which there were two possible outcomes, success or failure. Generalizing, suppose that a random experiment can have N + 1 possible outcomes; outcome i occurs with probability /?,-. Thus

Po + + PN = If RND is uniformly distributed in the unit interval [O, 1), then fraction/?0 of the RND vales will be in the interval [O, p0), fraction p1 of the RND vales will be in the interval [p0, p0 + pt), . . . . Thus to simlate an outcome i, use this procedure: Genrate RND, and determine which interval contains RND. More specifically, if RND < p0, set i = 0; f not, but RND < p0 + plt set / = 1; if not, but RND < p0 + p1 + p2, set / = 2; . . . . That is, the simulation results in outcome i if i is the smallest integer so that RND < p0 + + p. Note that such an i exists since p0 + + pN = 1 and RND [O, 1). Henceforth we will denote these probabilities using array notation. In the variable declarations part of the programs, P will be array [0..100] of real. (N is assumed to be * 100.) Thus in particular, P[I] denotes/?,. (See Fig. 7.1.)
Simulation with a Finite Number of Outcomes Determine N and the probabilities P[0], . . . , P[N]. Form the sums SUM[I] : = P[0] + + Pfl] for I : = O to N. Genrate RND and find the smallest valu of I so that RND < SUM[IJ.

Sec. 7.3

Simulating Discrete Distributions

191

PO]

P2]

P[3]
( \

1 0

SumfO]

1 Sum [1] Sum [2] i

RND

i
Sum [/V - 1]

i 1

Sum [/V] =

Sum [/] : =-P[0] + + P[I]

Figure 7.1

Example 8 A random experiment can have outcome O, 1, . . . , or N. Write a program that can input MAX (the number of simulations to perform), N, and the N + 1 probabilities P[0], . . ., P[N]. Output all MAX many simulations of the experimentSolution Both P and SUM are arrays. A while loop is used to find the smallest I so that RND < SUM[I].
const NORM = 10000; ADOR = 4857; MULT = 8601; &~~ - ~ varl,SEED,MAX,J,N: nteger; RND, SUM: real; P,SUM: array [0..100J of real; begin readln (MAX,N); for J:= O t o N do read(P[J]); SUMI] := P[0]; for J:= 1 t o N doSUM[J]:= SUM[J-1] + P[J]; SEED:= 0; for J:= 1 to MAX do begin

SEE"D:= (MULT*SEED + ADOR) mod NORM;

RND:= SEED/NORM;
l:= 0;

while RND > = SUM[I] do l:= 1 + 1; write(l);

end; end.

Example 9 Use the method above to simlate 100 rolls of a fair die.

192

Interlude: Modeling Randomness

Chap.1

Solution Each outcome has equal probability 1/6. Henee they do not have to be input into the program. Neither does SUM[I] have to be stored since SUM[I] = 1/6. Why?
const NORM = 10000; ADOR = 4857; MULT = 8601; **=--. -,6.01 var I,SEED,J; integer; RND: real; begin SEED:= 0;
for J: = 1 to 100 do begin SEED:= (MULT*SEED + ADOR) mod NORM; RND:= SEED/NORM; l:= 1;

while RND >= 1/6 do l:= 1 + 1; write(l); end;

end.

Note that I is initialized to 1 before the while loop rather than O as in Example 8. Why? Example 10: Simulaing a Binomial Random Variable Input MAX,N,P and simlate MAX many experiments; each consists of performing N Bernoulli triis with probability P of success and determining the number of successes on all N triis. Solution There are two methods. The first is to simlate a total of MAX N many Bernoulli triis, split them into groups of N each; for each group of N, count the total number of successes. (Problem 7.13 asks that you do this.) The second method is to compute the binomial probabilities
pl .

and the cumulative sums as in the boxed method above. The probabilities P[I] will be computed recursively using

N - I+1 P P[I - 1] I Q

Sec. 7.3

Simulating Discrete Distributions

193

as in Chapter 4, Example 17. The body of the program is

begin readln(MAX,N,P);
Q:= 1-P; P[0]:= Qjf*N;

for J:= 1 toN doP[J]:= ((N-J + 1)/J)(P/Q)*P[J-11; SUMI] := P[0]; for J:= 1 to N do SUM[J]:= SUMlJ-1] + P[J]; SEED:= 0; for J:= 1 to MAX do begin SEED:= (MULT*SEED + ADOR) mod NORM; RND:= SEED/NORM;
l:= 0;

whileRND>= SUM[l]dol:= 1 + 1; write(l); end;

end.

Let X be a random variable with range { 0 , 1 , . . . , N} and P(X = i) = p.. Let us see how the foregoing method for simulating a valu of X appears graph'ically. The distribution function for X is F(t) = P(X ^ t) as a function of t. Since X is discrete, F is a step function with jumps at the integer points i = O, 1, . . . , N. To simlate a valu of X, the method is to compute RND and find the smallest z so that RND < pQ + + p. Graphically, the valu RND is plotted on the y axis; a horizontal line is drawn from RND; where it intersects the steps determinen the simulated valu i. (See Fig. 7.2.) More generally, if the discrete random variable X has range {x0, xl, . . ., x N j with the vales increasing (x < xi+l), then the distribution function of X is a step function; if p, = P(X = x), then F(x) = P(X ^ x) = EPLY = x) = 2 pt

1 -

RND

_T
o
N Figure 7.2

194

Interlude: Modeling Randomness

Chap. 7

RND3

RND,

RND,

Figure 7.3 RND, simlales xa; RND2 simlales x3; RND3 simlales XN.

where the sums extend over all ndices / with x ^ x. Graphically, the distribution function appears as in Fig. 7.3. Then the graphical method for simulating a valu of X is this: First genrate RND, lcate RND on the y axis, and find the x valu exactly below the intersection point of the horizontal line through RND and the graph of F.

PROBLEMS 7.11. A box contains 4 red and 3 blue pencils. Three are selected at random without replacing any of them. Let X be the number of red pencils in the sample. What is P(X = i) for / = O, 1, 2, 3? Write a program to simlate and print 100 samples. That is, output should appear as a list of integers

2 3 O 2 O 1 1

...

indicating "2 red in first sample, 3 red in second sample, O in third sample, . . . ." Input should be P[I] for / = O, 1, 2, 3. 7.12. Input MAX and P and simlate MAX rhany experiments. Each consists of flipping a coin twice with probability P of heads. Outcome O corresponds to O heads (TT) with probability Q2 = (1 P)2; outcome 1 corresponds to HT with probability P-Q; outcome 2 corresponds to TH; and outcome 3 corresponds to HH. 7.13. Input MAX, N, and P and simlate MAX many experiments; each consists in performing N Bernoulli triis as in Example 10. Do this by performing MAX-N many smulations of Bernoulli triis. An outline of the program is
SEED:= 0; for J:= 1 toMAXdo begin SUM:= 0; forK:= 1 toNdo begin genrate RND and increment SUM if RND < P end; write(SUM); end;

Chap. 7

Problems

195

-7.14. (Continuation) Have your program print the average number of successes for MAX many simulations. This should be cise to the expectation N-P.

7.4 SIMULATING DISCRETE DISTRIBUTIONS WITH A COUNTABLY INFINITE NUMBER OF OUTCOMES

Assume throughout this section that random variable X has range {O, 1, 2, . . .] Let P[I] = P(X = /). To simlate a valu for X requires splitting the unit interval [0,1] into countably many subintervals, then selecting uniformly distributed RND. and finally determining in which subinterval RND is located. Using the techniques of the preceding section would require first finding the infinitely many sums
SUMtl] := P[0] + + P[l]

for I : = O to oo as in Fig. 7.4. Since these infinitely many operations are impossibk on a computer, we need an alternative. One possibility is to find a valu M so that SUM[M] is nearly 1. If the valu of RND is less than this valu, the simulation should produce the smallest I so that RND < SUM[I] as before; if, however, RND 2s SUM[M], the simulation will produce outcome M + l. An alternative and the one used here is to compute successively vales of SUM[I], stopping when RND < SUM[I] for the first time. This method is less efficient than the first method since with many simulations the valu SUM[0]. SUM[1], . . . must be calculated many times. On the other hand, there is no top cutoff valu as the upper bound for the simulated valu of X. One can improve the method by storing SUM[0], SUM[1], . . . as they are computed. Then when the next RND is generated, one checks to see if RND is less than any of sums SUM[I] computed so far; if not, more vales of SUM[I] are computed (and storec until a valu of I is found with RND < SUM[I] . This improvement is worthwhile but for the sake of clarity will not be incorporated in the examples (see Problem 7.18). Example 11: Simulating a Geometric Random Variable Recall that if W is the waiting time until the first success in a succession of BernoulM triis, then
P(X = f) = qi'^p

for/ = 1, 2, 3, ... Write a program to input MAX (number of simulations) and P and output the MAX many waiting times j simulated. Solution The recursive formula P(X = /) = qi^p =
P[Q]

-2p = qP(X = / - ! )

Sum [0]

Sum[1]

Sum [2]

Sum [3]

Sum [4]

Figure 7.4

196

Interlude: Modeling Randomness

Chap. 7

1P
1 Figure 7.5

is used (but see Problem 7.17 for an alternative method). These probabilities split the interval [O, 1] as in Fig. 7.5.
const NORM = 10000; ADOR = 4857; MULT = 8601; ^ var I,SEED,J,MAX: nteger; RND,SUM,PROB,P,Q: real; begin readln(MAX,P); Q:= 1-P; SEED:= 0; for J:= 1 to MAX do begin SEED:= (MULT*SEED + ADOR) mod NORM; RND:= SEED/NORM; PROB:= P; SUM:= P; l:= 1; whileRND> = SUMdo begin PROB:= Q*PROB; SUM:= SUM + PROB; l:= 1 + 1; end; write(l); end; end.

Example 12: Simulating a Poisson Random Variable Input should be MAX for the number of simulations and L for the parame ter X. Output should be MAX many simulations of a Poisson distributed random variable X with parameter L. Solution Recall (Section 6.1) that the Poisson probabilities satisfy this recursive formula:

P(X = i) = ~P(X = i- 1)
for i = 1, 2, . . ., where L is the parameter. The program is similar to that of Example 11. There are only two differences: the initialization
PROB:= P(X = 0) = Sec. 7.4 Simulating Discrete Distributions
197

and the modification

PROB : = UPROB/I using the recursive formula. The body of the program is
begin readln(MAX,L); SEED: = 0; for J:= 1 to MAX do begin genrate RND; PROB:= exp(-L); SUM:= PROB; l:= 0;

do whleRND> = begin l:= 1 + 1; PROB:= L*PROB/I; SUM:= SUM + PROB; end;

write(l); end; end.

PROBLEMS 7.15. Let Xbe Poisson distributed. E(X) = K, Write a program that will input L (for X.) and simlate 1000 vales of X. After each series of 100 simulations, print out the sum of all the simulations of X so far generated and the ratio of this number to the number of simulations. This number should be cise to the expectation X. Run the program for several vales of L. 7.16. Let X be geometrically distributed. What is E(X)1 Write a program that will input P and simlate 1000 vales of X. Print out the average of all 1000 X vales. Run the program for several vales of P. 7.17. X is geometrically distributed. Recall (Section 4.3) that P(X =/) = !- q'. Use this method (rather than the one in Example 11) to write a program that will input P and positive integer K. Perform 1000 simulations and print the number of times X exceeds K. (This should be approximately WOOP(X > K) = OOOqK.) 7.18. Incorprate the improvement discussed before Example 11 into that example. Declare another variable TOP (integer) whose valu is the largest I so that SUM[I] has been computed. After RND is generated, check to see whether RND < SUMfTOP]. If not, compute SUM[I] for I > TOP until RND < SUM[I]; then set TOP equal to I.
198

Interlude: Modeling Randomness

Chap. 7

7.5 SIMULATING CONTINUOUS DISTRIBUTIONS

The methods developed so far are useless for simulating a continuous random variable X. To see how to do this, we approximate X by a discrete random variable Fe which is defined to be within e of X regardless o the actual valu of X. More specifically, let e > O be given. Given the valu x o the random variable X, there is a unique integer j so that
(y - 1) e < * = /

For example, if e = .1 and x = 4.57, then 45(.l) < x ^ 46(.l) implies that y = 46; with e = .1 and x = -37.208, then -373(.l) < x -372(.l) implies that y = -372. Given X = x, the random variable Fe is defined to be the unique valu of y'e so that (y - 1) e < x ^ y'e. More simply stated, given a valu x of the random variable X, Fe is defined to be x rounded up to the nearest mltiple of e. Note these properties of the discrete approximation Fe to X:
1- YE - e < X s FE. 2. Ys = X if Xis an integer mltiple of e. 3. The range of Fe = { . . . , -3e, -2e, -e, O, e, 2e, 3e, . . .}.

Let F and Fe be the distribution functions for X and Fe, respectively. At mltiples of e, F and Ff are equal: Fe(y'e) = P(Fe = y'e) = P(X =s y'e) = F(y'e) since Fe = y'e if and only if X *s y'e. Also since Jf FE, if Fe ^ , then X ^ t. Thus set inclusin holds between these events:
{FE ^ t} ^ {X ** t}

Thus for any valu of t,

Since Fe is discrete, its distribution function is a step function with jumps at points in its range. Fig 7.6 summarizes these facts in graphs of the distribution functions: Note that as e O, the distribution function Fe will approximate Fmore and more closely. To simlate a valu of X, do this: Choose a valu e > O, simlate a valu of ye; this can be done using the methods of the preceding sections because Fe is discrete. Then an approximation to a simulated valu of X will be t. Although this appears complicated, in the limit as e > O, the method turns out to be very straightforward. How is a valu simulated for Fe? Since Fe is discrete the method before Fig. 7.3 can be used. First genrate RND, find RND on the y axis, then draw a horizontal line through RND; the point on the x axis below the intersection point of the horizontal line and the distribution function for Y, is the simulated valu of
Sec. 7.5 Simulating Continuous Distributions
199

Figure 7.6 To approximate a simulation for random variable X with distribution F, simlate Yt with distribution
-4e

-3e

-2e

Ye. No tice what happens in the limit as e 0: The distribution function for Ye approaches the distribution function for X. Consequently, we can dispense with the discrete random variable Ye and summarize the technique so far: To simlate a valu of X, graph its distribution function F, genrate RND, find RND on the y axis, and find the point just below the intersection of the horizontal line through RND and the graph of F as in Fig. 7.7. In fact, the graphical device can be simplified further. If RND produces simulated valu x as in the diagram, then x must be that point which satisfies F(x) = RND. That is, once RND is generated, the simulated valu of ^Tis F^1 (RND). Recall from Chapter 5 that X is concentrated on the set / if / is the interval on which F = density/is nonzero. / is the set on which Fis increasing. (The slope of F is O on 7C.) Thus the inverse F"1 of F is a function from (O, 1) to /. The following method summarizes the theory developed in this section: Simulating a Continuous Random Variable Let X have distribution function F. Let F~l be the inverse of F on the interval of concentration /. To simlate a valu of X: 1. First genrate RND. 2.F'1 (RND) is the simulated valu.

Example 13: Simulating a Uniform Random Variable on [a, b] Let U be uniformly distributed on [a, b]. The distribution function is

"O

x a
a = x =s b
b ss x
Chap. 7

F(x) =

x -a
b -a
d

200

Interlude: Modeling Randomness

Figure 7.7 In the limit as e - O, simulated vales of Y, approach a simulated valu of X.

Thus the interval of concentration is the interval on which F is increasing, this is [a, b]. Written in terms of the dependent variable y, the expression for F(x) is x -a y = b -a To find the inverse, find x in terms of y:

x - a = (b - a)y x = (b a)y + a
This function is an expression for F~1. Thus to simlate a valu for A'first genrate RND, then set X = (b - a) RND + a. Note that if X is uniformly distributed on the unit interval [O, 1], then a - O and b = 1. Thus the simulated valu of X is (1 - 0) RND + O = RND, which certainly is intuitive since RND itself is assumed to be uniformly distributed on the unit interval. Review: Finding F'1 1. First write y - F(x) as an expression in x. 2. Solve for x in terms of y. 3. This new expression is the formula for F'1.

ExampJe 14 Write a program to simlate 100 vales of X, where X has density

1<x

Also, simlate E(X).Sec. 7.5 Simulating Continuous Distributions

201

Solution The interval of concentration is the interval on which/is nonzero. This is (1, ). For x e (1, ) the distribution function is

Inverting, we have

x =

Thus

The body of the program is

begin SEED:= 0; SUIV:= 0; for l:= 1 to 100 do begin SEED:= (MULT*SEED + ADOR) mod NORM; RND:= SEED/NORM;
X:= write(X:10:4)

SUM:= SUM + X;
end;

writeln; writelnCEXPECTED VALU ', SUM/100:12:4);

end.

Example 15: Simulating an Exponentially Distributed Random Variable Let X have exponential density with parameter X.:

o
p y = I \e Jo

x <G 0< x

Then the interval on which the density is nonzero is (O, ). For x > O,
X!

dt = 1 - e **

x = 202

Interlude: Modeling Randomness

Chap. 7

Henee the transformation from uniformly selected RND to X is

x-= ~ ln(1 " RND)

Otherwise, the program is identical to that of Example 14. Note that 1 - RND is also a random number in [O, 1]. Thus to simplify the calculations, one might simply set
_ -In (RND) X

There is one small technical point, though. Since one of the vales of SEED is O, RND = SEED/NORM will also be 0. In (RND) would then result in a program interrupt since In (0) must be computed. This does not happen with In (1 - RND) since RND [O, 1). Example 16: Buffon Needle Experment Recall Example 17 of Chapter 5, in which a needle 1 unit long is dropped onto a grid of parallel lines 1 unit apart. We saw that'
2 /*(needle crosses line) =
1T

Write a program using this result to determine experimentally the valu of TT. Solution These two random variables were defined in that example: X = distance of foot of needle to line to left <> = angle needle subtends with positive horizontal axis Then X is uniformly distributed on [O, 1) and <> is uniformly distributed on [O, ir). A crossing occurs if and only if
O
TT

and

eos s > 1 x

or
<S
1T

and

- eos s > x

where x is the valu of X and s is the valu of 4>. RND simlales X, but ir RND simlales <> since <S> is uniformly distributed on [O, TT). (Use Example 13 with a = O, b = TT.) With MAX equal to the total number of needle tosses, SUM is initialized to 0. SUM is incremented by 1 each time vales of JTand <I> are simulated which satisfy the conditions for a crossing. The body of the program is
begin
SEED:= 0; SUM:= 0; Pl:= 311415927; readln(MAX);

Sec. 7.5

Simulating Continuous Distributions

203

for l:= 1 to MAX do begin SEED:= (MULT*SEED + ADOR) mod NORM; X:= SEED/NORM;

SEED:= (MULT*SEED + ADOR) mod NORM; S:= PI*SEED/NORM; if S < = PI/2 then f cos(S) > 1-XthenSUM:= SUM + 1; if S> PI/2 then if -cos(S) > X then SUM:= SUM + 1; end; writeln(SUM/MAX:10:4); end.

The simulated valu of 2/ir is SUM/MAX; therefore, for large vales of MAX, TT = 2*MAX/SUM. Here are the results for several runs:

MAX 100 500

1,000 5,000 10,000

2*MAX/SUM 3.9216 3.3223 3.2949 3.1250 3.1358

PROBLEMS 7.19. Let X have density function

,, . _ ~~

x < O or 1 < x 0<x<l

What is the distribution function F? On what interval is it concentrated? What is F"1? Write a program to simlate 500 vales of X and print the number that fall into each of the intervals [O, .1), [.1, .2), . . . [.9, 1]. Campare these with the probabilities calculated using the density function. 7.20. Let X have density function
x <O

(x + I)3

0< x

What are the distribution function F, the interval of concentration, and F"1? Determine constants xl, x2, x3 so that X falls into each of the intervals

(O, x,)
204

(X,, x2)

(x2, x3)

(x3, oo)
Chap. 7

Interlude: Modeling Randomness

7.21.

7.22. 7.23.

7.24. 7.25.

with probability 1/4. Simlate 200 vales of X and print out the number that fall into each of these four categories. Let X be exponentially distributed with parameter L = \. E(X) = IIL. Write a program to input MAX, simlate MAX many vales of X, and print their average. Is it cise to 1/L? Run the program for several vales of L. (The Cauchy Distribution) Write a program to simlate the Cauchy distribution of Problem 5.45. Inputs should be MAX (number of simulations) and R. Simlate the janitor problem (Problems 5.42 and 5.43). Input MAX (number of simulations) and a time U. Output should be the fraction of the number of simulations in which the lights were on at least time U. Run the program for U = 1/4, 1/2 and compare each with the answer to Problem 5.43. (Continuation) Let V be the time that the lights are on. Compute. E(V) from the results of Problem 5.42. Run 1000 simulations of V and output the average. Assume that the number of accidents along a certain stretch of highway is Poisson distributed with parameter (average) X. X depends, however, on the visibility; assume that the visibility V ranges from 1 (clear daylight) to a mnimum of .2 (foggy night). Assume that the relation between the average number of accidents X and V is
X = 2V~l

where V itself is uniformly distributed on the interval f.2, 1]. Write a program to input MAX = number of simulations and output MAX many simulations of the number of accidents. For each simulation the valu of V must be simulated and then using X = 2IV the number of accidents is simulated and output. Also have the average number of accidents output.

Chap. 7

Problems

205 \

8,
History abhors determinism, but cannot tolrate chance. Bernard DeVoto, 1897-1955, The Course ofEmpire*

Chapter 8
Joint Probability Distributions

In most of the models developed so far the main interest was in how one random variable was distributed. The object of this chapter is to present methods for dealing with random experiments in which more than one quantity is random. For example, three random variables associated with the weather are the temperature, humidity, and wind speed. There are certainly relations among them; thus very cold temperatures are often accompanied by high winds. But the relation is not completely deterministic; there is only a tendency for high winds and low temperatures to be related. The exact relationship itself incorporates random features that may be more complex than are the individual distributions of the random variables. The clue to understanding such situations in which more than one random variable is involved takes us to the very definitions of sample space and random variable. Suppose that X and Y are both random variables on the sample space fi. This means that X and Y are both functions
X, Y: l - R

from l to the real numbers R. To find how X and Y are related, look at how the vales X(u>) and Y(co) are related for each sample point w. First discrete random variables are considered, then continuous ones.
*From The Course of Empire by Bernard DeVoto. Copyright 1952 by Bernard DeVoto. Copyright renewed 1980 by Avis DeVoto. Reprinted by permission of Houghton Mifflin Company.

206

8.1 JOINT DISTRIBUTIONS OF DISCRETE RANDOM VARIABLES We saw in Chapter 4 how tables are useful in understanding the distribution of a discrete random variable. They are equally useful in dealing with joint distributions. Example 1 Flip a fair coin three times. Let X equal the number of heads and Y equal the number of the flip on which the first head occurred (or O if no head occurs). Then here is a probability table with the sample points as first row:
HHH 3 1 1/8 HHT 2 1 1/8 HTH 2 1 1/8 THH 2 2 1/8 TTH 1 3 1/8 THT 1 2 1/8 HTT 1 1 1/8

Outcome

TTT 0 0 1/8

X
Y
Probability

From this table we can construct a table with vales of X along the top, vales of Y along the left, and the probabilities as entries:

1/8

0
1/8 1/8 1/8

0
1/4 1/8

0
1/8

0 0 0

0 0

This table is a complete summary of all the probabilities

P(X = x and Y = y)
over all the vales x in the range of X and the vales y in the range of Y. example, P(X = 2 and Y = 1) = 1/4. Example 2 For

From the table in Example 1, find P(X 2 and Y = 1) P(X 2 and y 1) P(X 2 or
Sec. 8.1

7 as 1)
207

Joint Distributions of Discrete Random Variables

Solution We simply consult the table:

P(X ^2

and

Y = 1) = P(X = O, 1, or 2 and

Y = 1)

2 and Y =s 1) = ]? P(A" = i and Y = y)

i,j

where the sum extends over all i from O to 2 and y from O to 1 . Henee

P(X 2 and Y 1) = |
Finally,

2 or

Y =s 1) = P(X 2) + P(Y = 1) _ 7 5 _ 1 ~ 8+ 8 ~ 2 = 1

and

Y = 1)

Note that this last probability can also be computed using the complementary event:

P(X s= 2 or Y = 1) = 1 - P(X > 2 and Y > 1) = 1 - O

For two discrete random variables the probability table for the joint probability distribution has the vales of one along the top and the vales of the other along the left. There is straightforward procedure for obtaining the distribution of one of the random variables from the table. Suppose that the vales along the top correspond to random variable X and those along the left to random variable Y. Then
Pxy

= P(X = x

and

Y = y)

is the entry in column x and row y.

y

To obtain
y

P(X = x) = ^ P(X = x and Y = y) = ^ pxy

all the entries in column x are added. Similarly, to obtain P(Y = y} = lLP(X = * and
X

Y = y) = ^ pxy
X

all the entries in row y are added.

208

Joint Probability Distributions

Chap. 8

Example 3 Using the table of Example 1 with an extra row and column equal to the sum, we obtain x
0 1 2 3

1/8

0
1/8 1/8 1/8
3/8

0
1/4 1/8

1/8 1/2 1/4 1/8

1
2 3

0 0 0
1/8

1/8
0 0
1/8

0
3/8

Thus we see at a glance, for example, that P(X = 2) =

Note that the probabilities of the individual distributions for X and Y are written in the margins of the table. Given the joint probabilities
Pxy

= P(X = x

and Y = y)

the marginal distributions for X and Y are their individual distributions. These are obtained by

P(X = x) = 2 P(X = x and Y = y)

y

P(Y = y) = 2 P(X = x and Y = y)

Example 4 A bin contains 3 bolts, 4 washers, and 2 screws. Grab 3; let /? be the number of bolts and W be the number of washers. What is the probability table for B and W, and what are the marginal distributions? Solution The joint probabilities are computed using combinatorics. Thus there are a total of 3 + 4 + 2 = 9 tems. 3W4W2 O/U/U P(B = 0 and W = 1) =
9 3

Sec. 8.1

Joint Distributions of Discreta Random Variables

209

More generally,

P(B = b and W = w) =

3 -

The probability table with marginis is therefore

o
0

i
3/84
24/84

B
6/84 12/84 1/84 10/84
40/84 30/84

0
4/84 12/84 4/84
20/84

0 0 0
1/84

2 3

18/84

0 0
18/84

0
45/84

4/84

For example, the marginal probabilities imply that

P(B = 2) = 1?

5.2 INDEPENDENT DISCRETE RANDOM VARIABLES

Tn Chapter 3 the events A and 5 were defined to be independent if P(A n B) = P(A)P(B) The definition for independence of random variables is similar. The idea is that X and y are independent if any event defined in terms of X is independent of any event defined in terms of y. But events defined by discrete random variables can be decomposed into sums of probabilities of the form P(X = x) and P(Y = y). Defnition Discrete random variables X and y are independent if

P(X = x and y = y) = P(X = x)P(Y = y)

for all vales x in the range of X and y in the range of Y. Note: For X and y to be independent, the product rule above must be satisfied

210

Joint Probability Distributions

Chap. 8

for all vales x and y in the ranges of X and Y. If for just one pair x and y the equality fails, then X and Y are not independen!. Example 5 From a bin with 3 red and 6 nonred light bulbs, sample 2. Let X equal 1 if the first bulb is red; otherwise, X is 0. Similarly, Y is 1 if the second bulb is red; otherwise, Y is 0. Are X and Y independen!? Solution The question cannot be answered until we know whether the sampling is with or without replacement. With replacement implies that the bin has the same composition from selection to selection and so X and Y will be independen!. Sampling without replacement suggests that X and Y are not independen!. Quite arbitrarily, we choose vales 1 for both X and Y and compue P(X = 1 and y = 1) = /'(first and second are red)

36
On the other hand,

P(X = 1) =
and

P(Y = 1) = P(Y = 1 | X = 0)P(X = 0) + P(Y = 1| X =

_ 3 6 2 3 ~ 8 ' 9 + 8'9

= 1)

Thus

= 1)P(Y = 1) =
so Jf and F are not independen!.

= 1 and Y = 1)

Suppose that we have the joint probabiliy lable for X and Y. Since he marginis are the vales of P(X = x) and P(Y = y), and since the entries in the

Sec. 8.2

Independent Discrete Random Variables

211

table are P(X = x and Y = y), there is a very straightforward way to check whether X and Y are independent once the table has been constructed: Discrete random variables X and Y are independent if and only if each entry in the probability able is the product of the corresponding row and column marginis. Example 6 Suppose that the range of Xis {O, 1, 2} and the range of Fis {4, 5, 6}. With these probabilities in the table given, fill in the rest of the table under the assumption of ndependence.
0 4 5 r 6
1/6

X 1
a e k
1/3

2
b f h
1/3

d
1/6

Solution / = (1/3 + 1/3) = 1/3 since the marginal probabilities for X must add to 1. Henee d = O since the column under X = O must add to P(X = 0) = / = 1/3. Since d O, ndependence implies that g = O and consequently that e = O and / = 0. Now ij - 1/6, so / = 1/2. Thus k = h = 1/6. Finally, a = b = 1/6 since each of the column sums must be 1/3; this in turn implies that c = 1/2.
X 1
1/6

0 4 5 6
1/6

2
1/6 1/2

0
1/6
1/3

0
1/6
1/3

0
1/6
1/3

0
1/2

Example 7 (Continuation) Construct a probability table for Z = X + Y. Solution The range of Z is {4, 5, 6, 7, 8}. Collapsing those entries in the table above which yield the same valu for Z = X + Y implies that

Probability
212

1/6

1/6

1/3

1/6

1/6

Joint Probability Distributions

Chap. 8

Suppose that X and Y are independent, that X is Poisson distributed with parameter (mean) a, and that Y is Poisson distributed with parameter (3. Consider the random variable Z = X + Y. To find the distribution of Z, note first that X, Y, and consequently Z have range {O, 1, 2, . . .}. For Z to equal k, X must have some valu j where j = k\ Y must then have valu k j. Henee

p(z = k) =

=y

and

" = * - y)

--

where the last equality follows from the binomial formula. But this is precisely the Poisson distribution with parameter equal to a + p. If X and y are independent, each Poisson distributed one with parameter a and the other with parameter (3, then

Z =X +Y
is Poisson distributed with parameter equal to the sum of the parameters a + 3Example 8

Let X and Y be the number of requests to the Pascal and FORTRAN compilers, respectively, during 10 minutes at the school computer. Assume that X and Y are independent, each Poisson distributed. Suppose that the average number of Pascal requests is 2.3 and the average number of FORTRAN requests is 1.8 in 10 minutes. Find the probability that there were fewer than 4 requests of either type between 10:00 and 10:10. Find the probability of more than 7 requests between 10:00 and 10:20. Solution Since the parameter for the Poisson distribution is the average, the result above says that the total number of requests in 10 minutes is Poisson distributed with parameter
2.3 + 1.8 = 4.1

Henee
4I 4I ( 1 + 4.1 + -^ +
2 3

= .4142
Sec. 8.2 Independent Discrete Random Variables
213

The number of requests in 20 minutes is Poisson distributed with parameter

4.1 + 4.1 = 8.2

Henee P(more than 7 in 20 minutes) = 1 - e~s-2 ^ ~~^~ = .5746

PROBLEMS i8.1. Suppose that this is the joint probability table for X and Y:
X

.07 .12 .33

.02 .21 .05

.05 .03 .01

.09 0 .02

(a) Find P(X =s 3 and Y > 4); (b) P(X =s 3 or Y > 4). (c) What are the marginal distributions for X and Y? Find (d) P(X = 3); (e) P(Y > 4). 8.2. Suppose that this is the joint probability table for U and V:
V
2
0 3

.1 .1 .1 .1

0 .1 .1 .1

0 0

0 0 0

1
U
-: 2
3

.1
.1

.1

Find (a)P(U -2andV^2);(b)P(U^ -2 or V ^ 2). (c) What are the marginal distributions for U and V? Find (d) P(U ~ -2); (e) P(V & 2). (f) What is the distribution of U + VI 8.3. A drawer contains 5 white, 4 black, and 3 striped socks. Grab 4; let Xbe the number of striped and Y be the number of black socks. Construct the joint probability table for X and Y; include the marginal probabilities.

214

Joint Probability Distributions

Chap. 8

8.4. If X and Y have this as their joint probability table, fill in the missing entries:
X 5

6 .3

.1 .1 .3
.4

1
2

.2

8.5. Suppose that X and Y are independen! and that this is their joint probability table. Fill in the missing entries:
X 5

4 ' 0 .2

6 .6

1
2

.1

.4

8.6. Suppose that .Y and Y are independen! and that these are the distribution tables for X and Y:
X
Probability
0 .1 1 .1 2 .3 3 .2 4 .3

y
Probability

5 .2

6 .3

7 .2

8 .1

9 .2

What is the joint probability table? Find P(X Y < 14) and P(Y - X < 4). 8.7. Suppose that the number of pine saplings in a 100-square-foot rea is Poisson distributed with an average of 3.5; and assume that the number of maple saplings in the same rea is also Poisson distrbuted but with an average of 2.8. Find (a) P(a 100square-foot rea contains at least 5 saplings); (b) P(a 200-square-foot rea contains fewer than 4 saplings). 8.8. The number of junk mail letters received per day is Poisson distributed with an average of 4; the number of real letters is Poisson distributed with an average of 2. Assuming independence between the two types; find (a) F(no letters received on Monday); (b) P(at least 5 letters on Monday and/or Tuesday); (c) P(3 junk letters on Friday | total of 5 letters on Friday). 8.9. Cars enter a toll booth at the rate of 1.4 per minute and buses at the rate of .2 per

Chap. 8

Problems

215

minute. Assume that each is Poisson distributed. Find (a) P(at least 2 vehicles in 1 minute); (b) F(at least 3 vehicles in 2 minutes); (c) P(at least 3 cars | 5 vehicles in 3 minutes). 8.10. A total of n + m Bernoulli triis are performed withp = probability of success. Let X be the number of successes in the first n triis and Y be the number of successes in the last m triis. What are the distributions of X and Y? Are they independen!? Show that X + Y is binomially distributed, n + m triis.

8.3 JOINT DISTRIBUTIONS OF CONTINUOUS RANDOM VARIABLES

For one continuous random variable X the distribution is determined by the density function /. In terms of infinitesimals

P(x < X x + dx) =

dx

This is the concept that is generalized to obtain the joint distribution of more than one continuous random variable. Let X and Y be continuous random variables. The joint density function h(x, y) is a function of two variables that is defined by

P(x < X = x + dx and y < Y y + dy) = h(x, y) dx dy

Thus h(x, y) dx dy is the infinitesimal probability that (X, Y) falls in the infinitesimal rectangle of the xy plae in Fig. 8.1. In a real situation we need to calclate probabilities of the form

P(a < X ^ b and c < Y ^ )

that is, the probability that X resides in the interval (a, b] and Y falls in the interval (c, d\. Geometrically, this is the probability that the ordered pair (X, Y) falls in the rectangle with corners (a, c), (a, d), (b, c), and (b, d). To obtain this, we add infinitesimal probabilities and since there are "continuum many" of them, an integral sign is used:

P(a< X ^ b and c < Y ^ d) = f f h(x, y) dy dx

Ja Je

Note that regions of the xy plae where h(x, y) is zero must be regions in which the ordered pair (X, Y) cannot occur. Only where the density function is positive can the actual vales of (X, Y) be found. Example 9 Suppose that the joint density function of X and Y is fry2 LO O < x < 1 and O < y < 1 (x, y) not in the unit square
Joint Probability Distributions Chap. 8

h(x, y) =
216

(1, 1,6)
y + dy z = h(x, y)

(1,0,0),
(1,1,0)
x x + dx

Figure 8.1

Figure 8.2 The joint density h is nonzero only in the unit space.

The graph of h is sketched in Fig. 8.2; note that it is a surface in 3 dimensions since h is a function of the two variables x and y. Thus the actual vales of (X, Y) must reside in the square with sides (O, 0), (O, 1), (1, 0), and (1, 1). Find (a) P(II2 < X ^ 3/4 and O < Y =s 1/3); (b) P(II2 < X and 1/3 < Y); (c) P(II2 < X ^ 3/4). Solution Each of these can be found by integraton.
l (a) P - < 2 3 1\ - and O < Y 4 3/

P'4 fl/3 6xy2 dy dx Ji/2 Jo

T3/4 Jl/2 1/2 '3/4
y

16-27
and < Y = < X 1 and i ri

<Y

6xy2 dy dx

1/2

-
Sec. 8.3

52* 1/2 27

26-3
27-4

Joint Distributions of Continuous Random Variables

217

1 3 ] = P(-<X*- and 0 < y s s l \2 4

3/4 ri

1/2 Jo _?_ 16 I 4

6xy2 dy dx

Recall from Chapter 5 that the density function of a single random variable Xmust satisfy two properties: Since X must have some valu, the integral of the density function from > to + must be 1. Since all probabilities are nonnegative, the density function must be itself nonnegative. For exactly these reasons: The joint density function of two random variables X and Y must satisfy
/"oc /"ex:

1.1

h(x,y) dxdy = P(-y><X<<

and - o o < y < c ) =

2. h(x, y) ^ O for all x, y

Example 10 Verify that

4x
O

O < x < 1 and 1 < y

otherwise

is a joint density function. What is P(Y > 5)7 Solution Since exponentials are positive, h(x, y) is certainly nonnegative,

and f f

h(x, y) dx dy = f f 4 ^ dx dy
J I J O

J X J oc

Jl V

2 r \ ~

=1
Henee &(*, j) s a joint density function.
218

Joint Probability Distributions

Chap. 8

P(Y > 5) = P(0 < X < 1 and 5 < Y < o)

LL
J_

dx dy

25 In Chapter 5 we found the geometric approach of valu in dealing with one continuous random variable. Thus the probability that Xhas a valu in the interval (a, b] is the rea underneath the graph of the density function between a and b. A similar interpretation has already been indicated for the joint density of X and Y: Given a regin A in the xy plae, the probability that (X, Y) is in A is the integral of h(x, y) over A; that is, the probability is the volume under the graph of h(x, y) over the regin A in the xy plae as depicted in Fig. 8.3.
z = h(x, /)

Volume = P((X, Y) e A)

Figure 8.3

Because of the difficulty of graphing functions of two variables, however, the geometric interpretation is of limited use here. But it does at least show this: The probability is zero that the ordered pair (X, Y) assumes any one particular valu. This is so since this probability is the volume under h(x, y) over just one point; but the volume of this Une segment is 0. In terms of integris,
P(X = a and

Y = b} =

h(x, y) dx dy = O

Thus, just as in the single-random-variable case, we can be rather careless in distinguishing between inequalities and strict inequalities. are equivalent to Although this is not true for discrete random variables, it is true for continuous random variables. The joint density function h(x, y) encodes all information about the distribution of X and Y. Thus it should be possible to obtain the individual densities of X and Y from h. In the discrete case in the preceding section, the individual distributionsthe marginal distributionswere found by summing along rows or columns. Analogously, in the continuous case the individual densities are obtained

Sec. 8.3

Joint Distributions of Continuous Random Variables

219

by the continuos analog of summationintegration. Let us compute the distribution function F of X:

F() =
= P( - oo < X s= t and
= ['.

- oo < y < oo)

h(x, y) dy dx h(x, y) dy
dx

J ac J

-I

The density function is obtained by differentiating the distribution function. But the derivative of an integral (no matter how complicated the integrand) is the integrand. Therefore, the density function for X is

A similar integration yields the density function for Y. Let h(x, y) be the joint density function for X, Y. Then the individual or marginal density functions / for X and g for Y are obtained by integrating h(x, y) out with respect to one variable:

/(*) =

J =c

h(x, y) dy

g(y) = j_oc KX, y) dx

Example 11

For the joint density function of Example 10, find the distributions of X and of Y separately. Also find
Y>6

Solution With / the density for X and g the density for Y,

= 2x

for O < x < 1; f(x) = O for x < O or x> 1. Why? Similarly, g(y) = O for y <

220

Joint Probability Distributions

Chap. 8

1 and for _y > 1,

P(X<
But

Y>6\

P(X < 1/2 and Y > 6) P(Y > 6)

P\X<- ana Y> 6 =

dy dx =

144

Henee divisin yields

F>6 = 36 1 144 ^ 4

Example 12 Suppose that X, F have joint density function

c sin (x + y)
h(x, y) = <

O < x, y <
either x or y not in O,

Find the valu of the constant c. Find the individual densities of X and Y. Solution Since h(x, y) is nonzero only in the square O < x, y < ir/2,
ir/2 r-ir/2

c sin (x + y) dx dy

[-ccos(x + y}\l=llz dy
c eos y - c eos I + y I I dy

JO

= 2c

Therefore,

Sec. 8.3

Joint Distributions of Continuous Random Variables

221

But f(x) must be O for x < O or x > ir/2. For O < x < ir/2
rv/2 l f ( x ) = J0 2 sm
1

dy
TT

= - I eos x - eos I + x

= - (eos x + sin x)
In similar fashion the density function for Y could be computed. But here is a shortcut: Since the joint density function h(x, y) is symmetric in the variables x and y, the density function for Y has the same form as the density function for X with the roles of x and y reyersed:
O

00 H -(eos i

y + sin y)

y <Q or - < y 2 ,, O < y <

INDEPENDENT CONTINUOUS RANDOM VARIABLES

Suppose that ^f and y are continuous random variables with joint density function h(x, y); let/be the density function of Xandg the density function of y. Intuitively, X and y are independent if any event defined by X is independent of any event defined by Y. To see what this means in terms of the density functions, consider the event that X is in the interval [x0, x0 + A*] and Y is in the interval [y0, y0 + Ay]. Using the joint density, we obtain
po + Ajr ryo + A>>

and

Ay) =

Jxo

Jyo

h(x,y)dydx

for small A* and A_y. On the other hand, assuming that the events x0 + &x and y0 ^ Y y0 + Aj are independent,
P(x0 ^ X x0 + A* and y0 =5 Y = y0

; x0
Cxo + A x
f"yo + Ay

Y ^ y0 + Aj)
Jyo

f(x) dx
"
-*U

g(y) dy

= f(Xo) A* g(y0) ,

222

Joint Probability Distributions

Chap. 8

for small vales of A x and Ay. By equating these two expressions and canceling A* Ay, we motiva te this
Definition

Continuous random variables X and Y are independen! if their joint density function is the product of the individual density functions: h(x, y) = f(x)g(y)

Example 13 (Continuation of Examples 10 and 11)

X and Yare independent random variables because their joint density is the product of the individual densities: For O < x < 1 and 1 < y h(x,y) = 4 = 2 * =

There is a converse to the boxed formula above: //the joint density function can be factored h(x, y) = f(x)g(y) then X and Y are independent (although / and g are not necessarily the densities for X and Y). (See Problem 8.13 for the proof.)
Example 14

Let 7\ and T2 be the lifetimes of two components used sequentially. That is, assume that component 1 oprales for time 7\, after which component 2 is plugged in. Assume that the two lifetimes 7\, T2 are independent. Also assume that 7\ has the uniform distribution on [O, 2] while T2 has the exponential distribution with parameter 3. Find the probability that the second component will last at least 2.5 time units after the first component is plugged in. That is, find
P(T, + T2 ^ 2.5)

Solution The densities fv for 7\ and /2 for T2 are

<2
3x

otherwise

3e O

O< x x <O

Sec. 8.4

Independent Continuous Random Variables

223

\
x + y = 2.5

(2,0)
Figure 8.4 Figure 8.5

Since 7\ and T2 are independent, their joint density function is the product of the individual densities:

y O <x <2 and y > O

h(x,y) =
otherwise The joint density is nonzero in the regin of the xy plae shown in Fig. 8.4. The set of points (x, y) (where x represents the valu of Tl and y represents the valu of T2) satisfying x + y 5* 2.5 is the regin A in Fig. 8.5. Henee

h(x,y)dydx
dy dx
J2.52

O J2.S-X 2

o 2
- 1 .,-7.5 . 1 -3x12

*
1

3
-75 6

lo

6
Example 15

Suppose that X and Y are independent and have densities / and g, respectively,
Joint Probability Distributions Chap. 8

(O, 1)

(2,0)

Figure 8.6 The joint density is concentrated in the infinite strip.

where

Find P(AY > 1). Solution The joint density is concentrated in a strip.
O < x <2 and Ky

otherwise ^ Since x represents the valu of the random variable X and y represents the valu of Y, P(XY > 1) is the integral of h over the regin in the xy plae where xy > 1 (see Fig. 8.6). Henee

x =2

A v2 8
-

dy
x=l/y

//!>

y2 ~

8,,S

I ^

y
31 32
See. 8.4 Independen! Continuous Random Variables

225

Let A be a regin of the plae. To find P((X, Y) e A), first sketch the regin A and use it to find the limits of integration in the formula

P((X , Y) e A) =

h(x, y) dx dy

PROBLEMS 8.11. Suppose that X and Y have joint density function

1,1

\ _ \c(x + y) [O

O < x < 1 and O < y < 1 otherwise

(a) Find c. (b) Find the marginal densities. (c) Are X and y independent? Find (d) P(X 1/3); (e) P(Y 5= 1/2). 8.12. Suppose that X and Y have joint density function

O< x

and O < y

h(x, y)
O

otherwise

(a) Find c. (b) Find the marginal densities. (c) Are X and Y independent? Find (d) P(X > 2); (e) P(Y 3) 8.13. Suppose that X and Y have joint density function h(x, y) = f(x)g(y)

which can be written as a function of x times a function of y. Show that the densities of X and Y are, respectively,

cf(x)
where the constants c and d are
= P
J -

and

dg(y)

= "
J-

cd = 1. Conclude that Jf and y are independent. 8.14. Use the result of Problem 8.13 to check whether X and Y are independent if their joint density function s

1 < x and 1 < y

(a)

otherwise

y O < x < 1 and 1 < y < e

O otherwise

226

Joint Probability'Distributions

Chap. 8

ch the

1 < x < 2 and 1 < y < 2 oherwise

where c is a conslan

(d)

2x O <x < 1 and O < y < 1 O otherwise

Find (d)

For each part, also find the individual densities of X and Y. 8.15. Suppose that X and Y are independen! each with exponential distribution with parameter \ = 3. (a) What is the joint density function? Sketch the regions in the xy plae where (b) x + 2y = 2; (c) x + y = 2; (d) x - y ; = 2. Find (e) P(X + 2Y = 2); (f) P(X + Y ^ 2); (g) P(X - Y * 2). 8.16. Suppose that X and Y are independen!; X is uniformly distributed on [O, 1] and Y is uniformly distributed on [O, 2]. (a) What is the joint density function? Find (b) P(X + Y 2); (c) P(X - Y > 0.5); (d) P(XY < 1) by computing reas. 8.17. Suppose that X and Y are independen!, ^isexponentially distributed with parameter 5 and y has density

it? Find

Find (a) P(XY > 1); (b) P(Max {X, Y}<2). .(c) Sketch each regin in the xy plae.

: densities

5.5 THE SUM OF CONTINUOUS INDEPENDEN! RANDOM VARIABLES (OPTIONAL)

it if their

Let X and Y be the lifetimes of two components used sequentially: as soon as the first component fails, the second is installed. Under the assumption that X and Y are independen!, what is the distribution of the lifetime of the system consisting of the two components? In this section a formula for the density function of Z = X + Yis derived. As before, let / and g denote the density functions of X and Y, respectively. By independence the joint density is h(x, y) = f(x)g(y) To find the density for Z, first the distribution function is found and then differentiated. Given a fixed valu z, to find P(Z = z) = P(X + Y =s z) requires that the joint density function be integrated over the set A in the xy plae depicted in Fig. 8.7 where
A = {(x, y) : x + y = z}

Chap. 8

Sec. 8.5

The Sum of Continuous Independen! Random Variables

227

x + y =z

Figure 8.7 The shaded regin is {(x, y) : x + y z}.

Thus the distribution function for Z is

Fz(z) = P(X+

-Lt
J 30

h(x, y) dy dx
f(x)g(y} dy dx

-u
J X

= i' f(x)G(y)\'- dx
= i" f(x)G(z - x) dx

where G is the distribution function of Y; the second-to-last equality follows by the fact that the distribution function G is the integral of the density function g. To obtain the density function for X + Y, differentiate this last expression with respect to z. Interchanging the derivative and the integral and noting that the derivative of distribution function G is density function g implies this result: Let X and Y be independen! with density functions/and g. Then the density function for the sum Z = X + Y is given by the convolution of /and g: fz(z) = f(x)g(z - x) dx

Example 16

Suppose that a radio uses one battery. The original battery will last a random length of time X that is exponentially distributed with average lifetime 1 week. A spare is bought of hgher quality; once installed, the spare will last time Y that is exponentially distributed with average life 2 weeks. If the radio together with the original and the spare batteries are taken on a trip, what is the distribution of the total time X + Y until both batteries have died? Find P(radio lasts at least 4 weeks).
228

Joint Probability Distributions

Chap. 8

Solution Since the parameterfor the exponential distribution is the reciproca! of the average, the density functions / for X and g for Y are

ro

x <o
O< x

y <O

Henee the density function of the total lifetime X + Y of both batteries is (z) =

-x)d

Note that/(jc) = O for x < O and g(z - x) = O for z - x < 0. Thus the integrand f(x)g(z
for

- x) = O

x < O or z - x < O
That is,

x < O or * > z
Thus the limits in the integral defining h can be taken from x = O to x - z:
A(z) - JQ e
/I ("7\ I >

^ *

.5e
S*

-3(2

J^J

dx

//V

= .5e~-5* I e~-5x dx Jo

This is the density for Z = X + Y. P(radio lasts at least 4 weeks) = P(X + Y ^ 4)

= J4 f (e-5* = .2524

dz

If Jand Yare independent and both concentrated on (O, ), the convolution formula for the density of Z = X + Y is

fz(z) = jf(*)g(z - x) dx
Sec. 8.5 The Sum of Continuous Independent Random Variables
229

In Example 16 the case when the spare and the original both have the same distribution is of particular interest. In this case the lifetime of the radio is the sum of two independent, exponentially distributed random variables with the same parameter. But in Section 6.6 we showed by another approach that this sum is gamma distributed. Let us see how the convolution formula also leads to the gamma distribution. Example 17 Suppose that

f(x) = g(x) =

e?-**

x <O x >O

and that X and Y have densities / and g. Show that the density of S = X + Y is the gamma density with n = 2. Solution

h(z) = | J(x)g(z-x)dx

Ke

Xe

*' dx

= X2

f*

e-Kz dx

PROBLEMS 8.18. Suppose that X and Y are independent; X is exponentially distributed with parameter a and Y is exponentially distributed with parameter (3. Assume that a = (3. Find the density for X + Y. 8.19. (Continuation) Assume that the length of time until the light burns out is exponentially distributed with parameter a = 2 (average ufe 1/2 unit of time.) Assume that the fix-it time is also exponentially distributed with parameter P = 3 (1/3 unit of time until it is fixed, on the average). Find the probability that the light burns out and is fixed before time 1. 8.20. Show that the density for X + Y derived in Problem 8.18 approaches
fa2xe-""
O

x >O
O> x

as (3 > a. (Hint: Use L'HpitaFs rule.) Show that this confirms the formula for the sum of two independent exponentially distributed random variables with the same parameter derived in Example 17. 8.21. Suppose that U and V are independent and each is uniformly distributed on [O, 1].

230

Joint Probability Dstributions

Chap. 8

Let W = U + V. With/and g both equal to the uniform density on fO, 1], show that the density for W is
1 dx

for O < z < 1 for 1 < z < 2

h() =

f(x)g(z -x)dx =
1 dx

by considering the intervals on which/(;c) and g(z - x) are both nonzero. Compute the integris and sketch the density h. 8.22. (Continuation) Suppose that X and Y are independent each uniformly distributed on [O, 1]. Use the density computed in Problem 8.21 to find P(X + Y < 1), P(X + Y 3= 1.5), and check your answers by computng reas. 8.23. Suppose that X is exponentially distributed with parameter \ = 2 and Y is uniformly distributed on [O, 1]. What is the joint density of X and Yl Show that the density orX + Fis

/; f.
Compute these integris.

2C-2* dx

for O < z < 1

i
2e~2x dx for 1 < z

8.6 n INDEPENDENT RANDOM VARIABLES

Let Xl, . . . , Xn be n random variables. Generalizing the definition for two random variables, we say that they are independent if all possible events defined by them are independent:
Defnition X1, . . . , Xn are (mutually) independent if for all sets Al, . . . , An of real numbers

P(x, e Alt. . . , xn e An) = P(X1 e A,) P(xn e ^)

For discrete random variables a simpler characterization is possible: The sets A can be restricted to the particular singleton sets consisting of points in the range of X,: If X1, . . . , Xn are discrete, then independence means that P(X = Xl,...,Xn=xn) = P(X, = Xl) P(Xn = xn) for all possible vales xlt . . . ,x,,in the ranges of Xlt . . . ,

Sec. 8.6

n Independent Random Variables

231

Example 18

Perform n Bernoulli triis. Define the n random variables X1; . . ., Xn by 1 zth trial results in success [O zth trial results in failure Then Xly . . ., Xn are independent. This is intuitively obvious, but let's see how the definition above can be verified: Let j of the vales x be 1 and the other n j vales be 0. Then the event JY - Xi, v . . ,A yn - vVi-i Xni ) corresponds to a particular n-long sequence of S's and F's in which there are j successes and n - j failures. Henee the probability of this event is p'qn~'. On the other hand, P(X = x) is either p or q depending on whether x is 1 or O, respectively. Henee
P(Y Y \ PY Y \ nnn- * v"-J I/ *\~*~.n n) P q

since ;' factors are p and the other n j factors are q. Note that each X is a Bernoulli random variable and the sum 5 = X1 + + Xn is exactly the number of successes on all n triis; this is so because a success at trial z implies that X is 1 and henee 1 is added to S. In this way S literally counts the number of successes on all n triis. Thus 5 is binomially distributed. , But note this: One rarely verifies directly that random variables are independent. Usually, they are known to be independent by intuition. It is the consequencethe product rulethat is used. There is one property of independent random variables that has vast consequences. It is that seprate "groupings" of independent random variables are also independent. For example, if X1, . . . , X5 are independent, then XX2 + eos (X5)) and tan (X3 - ex) are also independent, however complicated their distributions. Here is a special case of the general proof: Suppose that X1, X2, X3 are independent and each is discrete. Let us show that X^ + X2 and X3 are independent using the characterization of independence for discrete random variables: P(Xl +X2 = z and X = w) = 2 P(X^ = x,X2 = z - x,X3 =

= S P(X = x)P(X2 = z - x)P(X3 = , = x)P(X2 *z-xj\P(X3=

J

= P(X1 + X2 = z)P(X23 = w)

232

Joint Probability Distributions

Chap. 8

where the sum after the first equality is over all vales x that X1 can take on; the second equality follows the defnition of independence for X^, X2 and X3. Assume that X1; . . . , Xn are independent. Let hv be a real-valued function of 7 variables and h2 be a real-valued function of k variables. Then the two random variables

are independent. (Only for notational convenience are the two groups the first y and the last k\ the same result holds for any two or more groupings of distinct independent random variables.) Example 19

Let
, y, z) = xy + z
h2(x, y) = sin xey

Then the result above implies that if X-_, . . . , X5 are independent, so are
Zt = + X3

and

Z2 = sin X4eXs

Of particular interest are independent random variables each of which has the same distribu tion. For example, in flipping a coin n times, if X is 1 or O depending on whether the z'th flip results in heads or tails (as in Example 18), not only are X1; . . . , Xn independent, but they are all distributed in the same way. Defnition Xlt . . . , Xn are said to be independent, identically distributed random variables (IIDRVs) if they are independent and each has the same distribution. Example 20 A sign consists of 10 light bulbs. Their lifetimes are independent, each with density (in terms of months)
-r

X> 1

/(*) =

Find (a) P(no bulb burns out before 2 months) and (b) P(only 2 bulbs left at 3 months).
Sec. 8.6 n Independent Random Variables
233

rn.Jt.9r -"-

Solution (a)If r,is the lifetimeof the th bulb, then 7\,. . . , 7\ 0 areIIDRVs.
P(T > 2 months) = dx = J2 x 2

ri

Thus, by independence, P(no bulb burns out before 2 months) = P(T > 2)10
1

(b)

p(r,<3) =

The number qf bulbs burning at 3 months is binomially distributed with p 1/3. (Each bulb is a Bernoulli trial.) Thus P(2 bulbs burning at 3 months) =
= .1951

In Section 8.2 we showed that the sum of two independen! Poisson random variables is again Poisson distributed with parameter equal to the sum of the individual parameters. Not surprisingly, this result generalizes: Suppose that X, . . . , Xn are independen! each Poisson distributed with parameters txj, . . . , an. Then

S = Xi + + Xn
is Poisson distributed with parameter
X. = ttj + + an

If the random variables are IIDRV, the parameter is

X = na

Proof. From Section 8.2 we know that X1 + X2 is Poisson distributed with parameter the sum a t + a2. Since Xlt X2, and X3 are independent, so are the two random variables X^ + Xz and X3. But each is Poisson distributed. Henee by the same result in Section 8.2,
X2 + X3 =
X

is Poisson distributed with parameter

aj + a2 + a3 =
+ ot2) + 3

Continu in this way (by induction in other words) until you obtain the sum of all n of the random variables.
234

Joint Probability Distributions

Chap. 8

Example 21 Assume that the number of shoplifters caught at a large department store is Poisson distributed with an average of .7 per day. Find P(none caught in a 6-day week) and P(at least 3 caught in 2 days). Solution Since the average is the parameter for the Poisson distribution, the parameter for each day is a = .7. Since the number caught on each of the 6 days constitute IIDRVs, the number caught in 6 days is Poisson distributed with parameter 6 .7 = 4.2. Henee

4 2 P(none caught in 6 days) = e'4 2 = e~4-2 = .0150

Similarly, P(at least 3 in 2 days) = 1 - P(0, 1, or 2 in 2 days)

1 + 1.4 = .1665

L42' 2

PROBLEMS 8.24. A store sells glazed doughnuts at the average rate of 1.3 per hour, chocolate at the rate of .6 per hour, and jelly filled at the rate of 2.8 per hour. Assume that the numbers of each sold are independen! and each is Poisson distributed. What is the distribution of the total number of doughnuts sold (a) in 1 hour; (b) in n hours? (c) Find P(at least 2 sold in 15 minutes). 8.25. (Continuation) (a) How many of each kind should the store stock if it wants to be 90% certain that it will have enough of each kind to meet the demand for 1 hour? (b) If it doesn't care what kind it has as long as there are enough doughnuts of any kind to meet demand, how many need to be stocked to be 90% certain to have enough for one hour? 8.26. In Chapter 6 we drived the gamma density. Do this with the techniques of the present chapter. Let
Sn=
+ T.

be the sum of n IIDRVs each exponentially distributed with parameter X. Show that Sn has the gamma density by mathematical induction: S, in fact is exponentially distributed and so is gamma distributed. Assume that the density of Sn is
x <O
/(*) =

(n - 1)!
Chap. 8 Problems

O <x

235

Show that Sn+l = Sn + Tn +l has density (for z > 0)

Conclude by induction that Sn + l is gamma distributed. 8.27. Two percent of the cereal boxes are crushed. A cartn has 24 boxes. There are 5 cartons in the back room. (a) What is the probability that a cartn has no crushed boxes? Let X = number of crushed boxes in cartn i for z = 1, . . ., 5. (b) What is the distribution of Xf Let 5 be the total number of crushed boxes in all the cartons; let T be the number of cartons that nave at least 1 crushed box. Find (c) P(S = 2); (d)P(r<2). 8.28. The office has 3 phones. The first rings at the rate of 1.2 times per 5 minutes, the second at the rate of .8 times per 5 minutes, and the third at the rate of 2.4 times in the same interval. Find (a) P(no calis in 2 minutes); (b) P(at least 3 calis in 4 minutes); (c) P(it was the first phone | exactly 1 cali in 3 minutes).

8.7 THE NONHOMOGENEOUS POISSON PROCESS

Recall that the parameter X in the Poisson process of Chapter 6 is the average number of calis per unit time. X was assumed to be constant. With the mathematical tools developed in this chapter, this assumption can be generalized. X = X() wl here be a function of t. For example, if X(7.1) = 2, the average number of calis per unit time ai time 7.1 is 2. We wl cali the function X = X() the intensity function. Suppose that calis arrive at the sales order department in such a way that these postlales are satisfied: 1'. Nonoverlapping intervals are independent; same as Postlate 1 in Section 6.3. 2'. Fix a time with O = < . For a small interval Ai the chance of a cali arriving in the interval (t, t + Ai] is approximately proportional to the length Ai: P(exactly 1 cali in (t, t + Ai]) Ai
M)

as Ai > 0. The difference between this postlate and Postlate 2 in Section 6.3 is that the average rate X is a. function of the time . 3'. The chance of more than 1 cali in an interval of length Ai is negligible for Ai = O, same Postlate 3 in Chapter 6.

236

Joint Probability Distributions

Chap. 8

X approximately constant

A()

Figure 8.8 The number of arrivals in the z'th interval is approximately Poisson distributed with parameter X(f,) Ai.

For example, the rate of incoming calis may taper off during the noon hour; it may rise to a peak at about 10:00 in the morning and around 2:00 in the afternoon. Or as another example, accidents along a highway ought to obey Postlales 1' to 3'. But the accident rate X() surely is a function of t: More accidents are likely during rush hours; fewer on the weekends. Another ame for the nonhomogeneous Poisson process is the Poisson process in a varying environment since the entire environment determines the actual valu of the intensity function X(). As in Chapter 6, let N, = #(calls arriving in the interval [O, t]) We know from that chapter that if X is constant, then N, is Poisson distributed with parameter Xi. We now want to derive the distribution of N, with X a function of /. To do this, split the entire interval [O, t] into subintervals each of size Ai. Let Ai be so small that the "intensity function" X() is approximately a constant on each of the subintervals as in Fig. 8.8. Since X() = X (/,-) on the zth subinterval, the number of calis arriving during this interval is approximately Poisson distributed with parameters X(,) Ai. Let the actual number of calis be N. The total number of calis in [O, t] is
N, = Nn

which is a sum of n independent Poisson random variables. Consequently, by the result of the preceding section, N, is also Poisson distributed with parameter equal to the sum of the parameters:
X(j) Ai + X(2) Ai
X(J Ai =

7=1

As the number of subintervals n > and the size of each subinterval Ai - O, the sum approximates more and more closely the integral
du

Sec. 8.7

The Nonhomogeneous Poisson Process

237

The Nonhomogeneous Poisson Process Suppose that the intensity function X = X() is a function of time. Nt = #(arrivals in [O, ]). Set Let

A(r) =
Then N, is Poisson distributed with parameter (and henee mean) A(): For P(Nt = /) = M^' e -A ()

Example 22 Assume that the intensity function X() = X is a constant. Show that the result above says that Nt is Poisson distributed with parameter X. Solution With X() = X a constant, the parameter for N, is

A(f) = I X(w) du = X
Example 23 Suppose that the intensity function is linear.

1 - O

0<t< 1 1<

(Thus calis can only arrive during the interval [O, 1] since X() is O thereafter.) Find the probability of (a) P(no calis in [O, .5]); (b) P(at least 1 cali in [O, <*>)); (c) P(call in [O, .5] | exactly 1 cali in [O, )). Solution (a) For the interval [O, .5], the parameter is
.5

Thus

(1 - w) du = u -

P(no cali in [O, .5]) =

= .6873

Figure 8.9

238

Joint Probability Distributions

Chap. 8

(b) For the interval [O, >), the parameter is

me. Let
Thus
: For

r r J X() du = j (l
o

- u)du = u -

2 u

1 2

P(at least 1 cali in [O, ,)) = 1 - P(no cali)

= 1 - e-1/2

= .3935 (c) Finally, P(call in [O, .5] | exactly 1 cali in [O, o)) t the result P(call in [O, .5] and no cali in (.5, 1]) P(exactly 1 cali in [O, 1]) (3/8)1 1! (1/2) ~
_ 3/8 = 1/2
_ 3 ' 4
1
1/2

(1/8) O! C

1/8

tiereafter.) O, )); (c)

Notice that the parameter for the interval (.5, ) is the same as the parameter for the interval [.5, 1], which is 1/8. Why?

PROBLEMS

| t 'r

(8.29J Suppose that N, is the number of calis for a Poisson process in an environment in which the intensity function is

Find (a) P(Nir/2 = 1); (b) P(N^ < 3); (c) P(moK than 2 calis in [O, >)). 8.30. Nr is the number of calis for a Poisson process in an environment in which the intensity function is

sin (i)

O < t < TT

TT<t

t +1
for t > 0. Find P(N, = j) for/ = 0,1,2, . . . .
Chap. 8
Chap. 8 Problems

What is (#,)?
239

-- Time f

Figure 8.10 x's mark failure times. B4,2 = s/(s + u).

8.8 THE BETA DISTRIBUTION (OPTIONAL)

Consider a renewal process as defined in Sections 6.5 and 6.6: Components are installed successively; the successive lifetimes 7\, T2, . . . form a sequence of IIDRVs each exponentially distributed wth parameter X(mean life 1/X). In Chapter 6 we showed that the total lifetime of the first n components
SH = 7\ + + Tn

is gamma distributed with parameters n, X. The density function of Sn is

O

u <O
M> O

(n - 1)!

Let n, m be fixed positive integers. Consider the random variable

B = Bnm = JL^n + m

Thus B is the fraction of the time that the first n components were operational over the total amount of time that the first n + m were operational. B is concentrated on the unit interval [O, 1] since O < Sn < Sn+m, The distribution of B = Bnm is called the beta distribution with parameters n and m. (see Fig. 8.10) It turns out that the distribution does not depend on the parameter X . (Can you see an intuitive reason why this is so?) Let us derive the density function / m for Bnm, Let n and m be positive integers. Let

be, respectively, the lifetime of the first n and the lifetime of the last m components in a succession of n + m component installations. Since {rjf^"1 are IIDRVs, Sn and Rm are independen!. Also, Sn is gamma distributed with parameters n and X, while Rm is gamma distributed with parameters m and X. Note also that Let O < < 1; the distribution function for B = Bnm is F() = P(Bnim ^ )

= P

240

Joint Probability Distributions

Chap. 8

= P(Sn ^ t(Sn + /?)) = P((l - t)Sn =s tRm)

Since Sn and Rm are independen!, their joint density function is h(x,y) = U O

otherwise

With x corresponding to Sn and y to Rm, F(t) is the integral of h over the regin depicted in Fig. 8.11.
(1 - t)x <ty

Figure 8.11

Consequently,
ty/(l -

, 30
p
ty/(l-t)

1)! Jo Jo ~ (n - 1)! (m - 1)

To obtain the density function/for B, differentiate this double integral with respect to t. Since the only part that is a function of t is the upper limit on x,
n+m

F'(t) =

r \ d 'y

(n - 1) ! (m - 1)! Jo \_dt Jo

dx

dy

\"+>r' tv y (n - 1)1 (m - 1)! Jo (1 - t)2 \1 - t

e-\tyl(\-t)ym-\e-\y

\"+m t"-1 (n - 1)! (m -1)! (1 - /)"+ 1 J

Note how the chain rule was used to obtain the second equality:
tQ

f(x)dX=g'(t)f(g(t))

when the upper limit of integration is itself a function of t. Returning to the

Sec. 8.8 The Beta Distribution (Optional)
241

computation of F'(t), substitute u = \y/( - t). Then y = (1 - f)u/\, dy = (1 - f)/\ du, so

</()

(n - 1)! (m - !)!'(! - f)"

+1 Jo

1 - , du
1

1.0

(n - 1)! (m - 1)!

e~u du
In fact

The valu of the last integral can be found using integration by parts.

Jo (n + m - 1)!

r* u>e~

r du = i \ u'"
Jo

e" du for / 2= 1. This implies that the above integral is

The beta density function with parameters n 3* 1 and m ^ 1 is

1 1 1 n -'l ) ! (u m' -^ Dv ! ^ -^' 0<<1

f < O or

1<

Fig. 8.12 depicts graphs of the beta density functions for various vales of n and m. Example 24 Find P(B3l = 1/2) and P(B6i2 3= 4/5). For the first, we are asking for the probability that the first three components last less than half the time ofour (3 + 1) components used sequentially. For the second, we are asking for the probability that the first six last at least four-fifths the total time of eight (6 + 2). Solution B3l has density

1\

f() =
forO<f<l. Therefore,

l
B6 2 has density

12

'

1/2

for O < t < 1. Therefore,

P
Note that when n = 1 and m = 1, then B is the fraction of the time un til 242 Joint Probability Distributions Chap. 8

dy = du

In fact sgral K

>f n and

Figure 8.12

The beta density functions.

the second burnout that the first component lasts. The density of B = Bll from above is
/() =

Oi! /0(1 - f) = 1

Thus Btl is uniformly distributed on the interval (O, 1). Note how this also follows from Section 6.4: Between time O and the time of the second component burnout, there was exactly one component (the first) burnout. When this occurred is uniformly distributed by the results of Chapter 6.

PROBLEMS 8.31. Five components are used successively. The lifetimes are exponentially distributed and independent each with parameter X. Find P(B32 =s f) for t = (a) 1/4; (b) 1/2; (c)3/4.
Chap. 8 Problems

243

8.32. Show that/,() = / (1 - ) and interpret this fact intuitively. 8.33. Let n + m = 5; plot/14, /2,3,/3,2, and/ 41 . 8.34. Show that the beta densities are normalized: Let
f1
n,m
I Jn,m\ '/

dt

Use an integration by parts to show that In<m = / n + i, m _i. Find 7 n + m _ l j l directly and conclude that / m = 1 for all n, m. 8.35. Find (5n,m): First show that E(Bnl) = n/(n + 1). (Does this make intuitive sense?) Use an integration by parts to show that
n n + 1

Use this recursive formula to conclude that

CYR

n
~

E(tin,m)

n + ln + 2
~5
+

~T

n +m - 2 n +m - 1

n n +m

8.9 RANDOM COMMENTS AND PROBLEMS: CONDITIONAL DENSITIES (OPTIONAL)

Suppose that X and Y are continuous random variables, but not necessarily independent. Then the specific valu of Y, once known, helps to predict the valu of X. That is, the density function of X can be revised given that Y is known to have a definite valu y. Throughout, assume that X and Y have joint density function h(x, y) and that f(x) and g(y) are the marginal densities for X and Y, respectively. To find the conditional density function for X given Y = y, compute
P(JC < Jf =s X + A x | y < Y s y

P(x < X ^ x + A x
fy + r f * ^
A +

and y < Y as y + A>Q

P(y <Y ^- y + Ay)

A(s, ) ds
Jy
JA:

dt

y +Ay
g(0 di

h(x, y) A * Ay

g(jO
AA;

830 where the approximation is valid for small AJ: and Ay. As Ay tends to zero, we
244 Joint Probability Distributions Chap. 8

can define the probability that X E (x, x + AJC] given Y = y approximately as

P(x < X =s x + A* | Y = y) s
As A* > the infinitesimal t, we obtain motivation for the following Definition The conditional density for X given Y = y is

h(x, y)

g(y)
defined at points y with g(y) = 0. Similarly, the conditional density for Y given X = x is

h(x, y)
defined at points x where f(x) 4= 0. Example 25 Suppose that the joint density function for X and Y is

* + y2)

O < x < 2 and O < v < 1

otherwise

Find the conditional density of X given y = v f o r O < j < l . Solution First find the marginal density for Y; for O < y < 1 ,

= \ (x,y}dx =
Therefore, for O < y < 1,

= (i +

h(x, y) _ 1 x + y2 g(y) ~21+y2

for vales of x such that O < x < 2. For x outside this interval, f(x\y) = 0. f(x\y) is only defined for O < y < 1. (It makes sense to seek P(event A\Y = y) only if y is actually a valu in the range of Y.) Note that if X and Y are independent, then h(x, y) = f(x)g(y), and consequently, X and Y independent implies that o zero, we
Sec. 8.9

= g(y)
Random Comments and Problems: Conditional Densities
245

Given a valu of Y, the conditional expectation of X can be defined using the conditional density o X given Y = y. In fact, by definition, E(X \ Y = y) = Similarly, xf(x\y) dx

E(Y \ X = x)
J X

yg(y\ \x)dy

Example 26 For O < y < 1 compute the conditional expectation of X given Y = y in Example 25. Solution

1 x(x + y2) 2(1 + y2) Jo 4 + 3y2 3(1 + y2)

PROBLEMS
i

Find the conditional densities of X given Y = y and y given X = x for the joint densities of the following. Also find E(X'\ Y = y) and E( Y \ X = x). 8.36. Problem 8.11. 8.37. Problem 8.14(c) and (d). 8.38. In Problem 8.36 use the expression for/(jc | y) to find P(X < 1/2 | Y = 1/3).

246

Joint Probability Distributions

Chap. 8

I shall never believe that God plays dice with the world. Albert Einstein, 1879-1955, Einstein: His Life and Times by Philipp Frank*

Chapter 9 Variances, Covariances, Correlation Coefficients, and More on Expectations

The expectation of a random variable is a number that summarizes its distribution. In gambling language the expectation is the fair entrance fee. But two games with the same expectation can be vastly different; in the first the stakes are high; thus the player can expect either to win much or lose much even though the expected winnings are the same as in the second game, where the stakes are low. The variance is a number that summarizes this variation. Two random variables may be independent. Or they may be very highly associated; that is, the actual valu of one may allow one to be fairly sure of the valu of the other. The correlation coefficient between two random variables is a single number that summarizes their degree of association. The key mnemonic tool for remembering (and helping to derive) the formulas to be developed is this: The expectation of a random variable is the sum of its vales times the probabilities. As we saw in Chapter 5, even continuous random variables obey this rule of thumb if one replaces sums by integris and probabilities by "infinitesimal probabilities" f(x) dx, where/is the density function.

*From P. Frank, Einstein: His Life and Times. Translated by George Rosen. Copyright Alfred A. Knopf, Inc. Reprinted by permission of the publisher.

247

9.1 EXPECTATIONS OFFUNCTIONS OF RANDOM VARIABLES

How is the expectation of a function of X computed? For examples, the expectations of sin X or ex or (9/5)X + 32? Suppose for definiteness that X is discrete; let h be a function and define the random variable Z by Z = h(X). Consider this gambling game: If the random experiment results in X = x, Martha wins $x and Ronnie wins $h(x). The fair entrance fee for Martha is E(X), but what is the fair entrance fee for Ronnie? He will win $h(x) with probability P(X = x). Thus the "sum of the vales time the probabilities" implies that he should pay 2 h(x)P(X = x), where the sum extends over the vales x in the range of X. Similarly, if a continuous random variable X assumes the valu x with infinitesimal probability f(x) dx, the random variable h(X) will assume the valu h(x). Thus the "integral of the vales times the infinitesimal probabilities" leads to the second formula: Denition Let h be a real-valued function of a real variable. 1. If X is discrete,

E(h(X)) = 2 h(x)P(X = x)
X

where the sum extends over all x in the range of X. 2. If X is continuous with density/,

E(h(X)) =

h(x)f(x) dx

Example 1 Suppose that X has density

O
/(*) =

O> x

or 2 < x

O<x <2

Fnd E(X3) and E(ex + 3). Solution Directly from the definition in the box,

_
E(e* + 3) =

= 3.2

(e* + 3) - dx

248

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

the expecis discrete; nsider this yins $jc and .t is the fair x). Thus should pay eoX. with infinvalue h(x). eads to the

= - \xex - e" + - x2 2 2 = \ (e2 + 7)

Note that if X is a continuous random variable and Z = h(X), then we now have two methods for finding E(Z). The first uses the formula in the box above. The second finds the density of the random variable Z using the methods of Section 5.7. That is, Z is itself a random variable whose density g(z) can be found and used to compute E(Z) directly. Thus
(Z) = f zg(z) dz
J -=0

That these two methods always yield the same valu of E(Z) is shown in Problem 9.13. Example 2 Let X be as in Example 1. Show that E(X3) = 3.2 by first finding the density function of Z = h(X) = X3. Solution Since Xis concentrated on [O, 2], Z = X3 is concentrated on [O, 8]. For 2 [O, 8],
Fz(z) = P(Z ^ z)

= P(X3 s* z)
= P(X ^ z1/3)

o
2/3

where the fourth equality uses the density function of X from Example 1. Differentiating the distribution function to find the density g for Z yields

z < O or

<z

*(*) = i
Therefore,

-1/3

O< z< 8

,-1/3

z-

= 3.2

in agreement with E(X3) found in Example 1. Chap. 9

Sec. 9.1

Expectations of Functions of Random Variables

249

Example 3 Let X be uniformly distributed on [a, b}. Find E(Xn) for positive integer n. Solution The density for X is \l(b - a) on the interval [a, b]. Thus
E(X") =
x" dx

1 f x" dx b - a Ja

n +1
Note that when n = 1,

6- a

which is the expectation of Jf found in Section 5.4 Example 4

Let X be exponentially distributed with parameter X. Find E(X") for positive integer n. Solution Using the boxed formula above, E(X") = Therefore, E(Xn) = \ une-u du
(*)
Jo

xn\e-^ dx

after the change of variable u = \x. Intgrate by pars with u' = e "andv = u": E(X") = I -**- X" I
u

du

The first term in brackets is zero since it is clearly O at the lower limit; and at ,

u" lim u"e~" = lim = O

using L'Hpital's rule for the second equality. Thus E(X") = u"e~" du = u"-le-u du X" Jo X" Jo Now factor out /X from the coefficient in front of the second integral to obtain

i r*

nr

L(X} iu F(X) - l( {~ u^e-du\

\ViFUo '

250

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

nteger n. Thus

The expression inside the parentheses is exactly E(X" l) since the expression is the right-hand side of equation (*) with n replaced by n - 1. Thus we have derived the recursion relation

E(X") =

n X

nn- l
X X

E(X"~2)

n n 1 n 2 ~ X X X

= -

~ X"

where we have used the fact that E(X1) = E(X) = 1/X for an exponentially distributed random variable. Consequently, if X is exponentially distributed with parameter X, then
E(X") = n

for positive

orn = 1, 2, 3, . . . .

Suppose

The special case of a linear function is of particular importance. that X is a random variable and

F = aX + b
where a and b are constants. Then F is a change of scale applied to X. If X represents an amount won in gambling, then for 6 = 0 and an appropriate choice of a, F represents the same amount in rubes. It seems clear that if one plays the game many times with dollars, computes the average winnings, and then converts hat average to rubes, this number should be the same as if one played many games with rubes rather than dollars from the start. This is to say that the expectation should be preserved. Expectation is preserved under a change of scale:

(*)

jnd-y = u":

:; and at <,

E(aX + b) = aE(X) + b

Note the special case when a = 0: The expectation of a constant is itself : to obtain

E(b) = b

Intuitively, if one mus win $b by playing the game, one can expect to win $6.
Chap. 9

Sec. 9.1

Expectations of Functions of Random Variables

251

Formal Proof of Foregoing Result. For definiteness, suppose that X has density function / (although a similar proof works if X is discrete). Then E(aX + b) = (ax + b)f(x) dx

= a

xf(x) dx + b f /(*) dx J -se

= aE(X) + b 1
since the first integral is by definition E(X) and the second integral is the total rea under the density function/, which is 1. How is the expectation of a function of more than one random variable computad? For examples, X + Y or X sin Y or the mximum of X and Y? Let g(x, y) be a real-valued function of two variables. [For example, to find E(X sin Y) the function g(x, y) = x sin y would be used.] Let h(x, y) be the joint density function of X and y. Using the heuristic idea that the expectation is the sum of the vales times the probabilities leads to:
Definition

1. If X, Y are discrete, then E(g(X, Y)) = 2 g(x,y)P(X = * and Y = y)

*,y

where the sum extends over all x, y in the ranges of X, Y, respectively. 2. If X, Y are continuous with joint density function h, then E(g(X, Y)) = " g(x, y)h(x, y) dx dy

Example 5

Suppose that the joint density function for X and Y is concentrated on the square with vrtices (O, 0), (O, 1), (1, 0), and (1, 1). For0^x,y ^ 1,

Find E(XY), E(X + Y), and E(X) using the joint density function. Solution
E(XY) =
252

xy

(*2 + y2) dx dy
Chap. 9

Variances, Covariances, Correlation Coefficients, and More

: X has den-

dy

3 3 + o \F 4>

dy

E(X + Y) =

y) (*2 + y2)

ic total rea >m variable md Y? Let ind E(X sin oint density the sum of

Example 6

Two cards are selected without replacement from a standard deck of 52. Let X = number of hearts and Y = number of spades. Find E(XY) and E(X + Y). Solution Let us first construct a probability table with the vales of X along the top and the vales of Y along the left. P(X = O and Y = 0) = P(no hearts or spades in 2 cards)

ctively.

= .2451 P(X = 1 and Y = 0) = P(l heart and no spades) '13\/13\/^

W
52 2 \

i the square

= .2549 Similarly, the other entries in the table are computed:

X

1
.0588

o y i
7

.2451
.2549 .0588

.2549

.1275

0 0

Chap. 9

Sec. 9.1

Expectations of Functions of Random Variables

253

Using the fact that an expectation is the sum of the vales times the probabilities: E(XY) = O O (.2451) + 1 O (.2549) + 2 O (.0588) + O 1 (.2549) + 1 1 (.1275) + O 2 (.0588)

= .1275
E(X + Y) = (O + 0) (.2451) + (1 + 0) (.2549) + (2 + 0) (.0588) + (O + 1) (.2549) + (1 + 1) (.1275) + (O + 2) (.0588)

= 1.000
[Can you see an intuitive reason why E(X + Y) = 1?] PROBLEMS 9.1. Suppose that X has density

/(*) =

O
2x

x < O OT 1 < x O <x < 1

Find (a) E(X), (b) E(X2}, (c) E(IIX), (d) E(sin X), and (e) E(aX + b), where a and b are constants. 9.2. (Continuation) Let U = X2, V = IIX, where Xis as in Problem 9.1. Use the methods of Section 5.7 to find the densities of U and V. Use them to find E(U) and E(V). Do they agree with the answers to Problem 9.1? 9.3. Suppose that Y has density
O

00 =

Ky

Find (a) (Y); (b) E(1/Y); (c) (1/Y2). 9.4. (Continuation) Let U = I/Y, V = I/Y2, where Y is as in Problem 9.3. Find the densities of U and V and use them to find E(U) and E(V). 9.5. Suppose that F and C are Fahrenheit and Celsius temperatures so that F = (9/5)C + 32. Assume that C has density
x

/(*) = -{ 7500 O

50 <x < 100

elsewhere

(a) Find (C). (b) Also find the density function of F and use it to find E(F). (c) Verify that E(F) = 9(C)/5 + 32. 9.6. Flip a fair coin three times. Let X = number of heads and Y = the number of the flip on which the first head appeared. (If no heads on all three flips, set Y = 0.) (a) Construct a probability table in which the vales of X are along the top and the vales of Y are along the left. Compute (b) E(XY), (c) E(X + Y), (d) E(X - Y), and (e) E(X) from the table.

254

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

&

9.7. A bin contains 5 apples, 4 bananas, and 3 cantaloupes. Let A be the number of apples and B the number of bananas in a sample of size 3 taken without replacement. (a) Construct a probability table in which the vales of A appear along the top and the vales of B appear along the left. Use the table to find (b) E(A + B); (c) E(A); (d) E(B); (e) E(AB). 9.8. Suppose that X and Y have joint density function

x +y
h(x, y) =
O

O < x, y < 1
elsewhere

Find (a) E(X), (b) E(Y), (c) E(XY), and (d) E(X + Y) using the joint density function. (e) Find the marginal density functions for X and Y and use them to find E(X) and E(Y) and verify the abo ve. 9.9. Suppose that X and Y have joint density function

IX
O

(xy + 1)

O < x, y < 1
otherwise

Find (a) E(X), (b) E(Y), and (c) E(Max {X, Y}), where Max {X, Y} is the mximum of X and Y. 9.10. Suppose that X has exponential density with parameter \. Find (a) E(hi(X))\ (b) E(h2(X)). h is the function

pe
LO

O< x < L
otherwise

where L is a fixed number. /z2 is the function h2(x) = mnimum of x and 1 Can you interpret these expectations? 9.11. Let a < b be fixed real numbers. Let h(x) be the "indicator function" of the interval [a, b]. That is,
0

x < a or b < x

h(x) =

a =s x ^ b
Show that

Assume that X has density function /.

E(h(X)) = P(a X 6) and interpret this result in gambling terms. Show that in general E(1/X) is not IIE(X) by considering this special case: Suppose that^f is uniformly distributed on the interval [1,2]. Find E(X), E(l/X), and compare E(VX) with VE(X). 9.13. Here is an outline of a proof that the two methods for finding E(h(X)) for .Ycontinuous result in the same valu in one important case: Assume that h is an increasing function. Let X have density fx and distribution Fx. The distribution of Z = h(X) is

Fz(z) = P(Z ^ z) =

Chap. 9

Problems

255

Differentiate to obtain the density of Z;

Therefore,

E(Z) =

zfz(z) dz

Fill in all the steps. 9.2 EXPECTATIONS OF SUMS AND INDEPENDEN!PRODUCTS

Suppose that X:, . . . , Xn are random variables and

o _ A y ~t _i_ . . . -I4- A. y J 1 n

For example, X might be the lifetime of the /th component in a renewal process and S would therefore be the total lifetime of all components used sequentially. Then the relation between E(S) and the individual E(X,)'s is The expectation of a sum is the sum of the expectations:

E(X1 + + Xn) = E(X) + + E(Xn)

Example 7 In Example 5 E(X) was computed to be 5/8. By symmetry (or a similar calculation), E(Y) = 5/8 also. Thus

E(X) + E(Y) = 5in agreement with the result found in Example 5.

Proof of the Foregoing Formula. Suppose that X and Y have joint density function h(x, y). Then
/"OD /"OC

E(X + Y) = J _ x J__ (x + y)h(x, y) dx dy

r-f r=c f-y

r=c

CT

r*

r~f roe

= I

xh(x, y) dx dy + I

yh(x, y) dx dy

r
J c e

ir
\ J x

h(x, y) dy\dx +
I

r y (ir
J 3 = V J = e

h(x, y) dx

\ dy
/

256

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

= I
J CC

xf(x) dx + I
J

yg(y) dy
CK

= E(X) + E(Y)
Note that the fourth equality follows from the fact that integrating h(x, y) with respect to y yields the density function for X, and similarly, integrating h(x, y) out with respect to x yields the marginal density function for Y. Now consider the case for general n: Represent the entire sum as the sum of two random variables and apply the result we have just proved:
E(Xl
Xn) = E((X,

!) + Xn) ) + E(Xn)
1 rather than with n to

= E(X1

and continu the process starting with the sum of n conclude that
Xn) =

E(Xn)
+ E(Xn)

wal process ;quentially.

which proves the result. The proof for discrete rather than continuous random variables is similar and only involves replacing integral signs by summation signs. The result is very useful; it allows calculation of E(S) in situations in which S has a complicated dstribution, but the Xs have simple distributions. Example 8 In Section 4.6 a binomially distributed random variable 5 was found to have expectation np, where n is the number of triis and p is the probability of success on any one trial. This result can be derived much more simply by decomposing S as a sum. Let (as in Chapter 8, Example 18)
X, =

lilar calcu-

1 O

z'th trial results in a success /th trial results in a failure

int density

in a succession of n Bernoulli triis. Then the number of successes on all n triis is exactly the sum
S = X1 + + Xn

since each X{ adds 1 to S if trial / resulted in success. Consequently,

E(S) = E(X1) + + E(Xn)

But each X is a Bernoulli random variable that assumes the valu 1 with probability p and the valu O with probability q. Thus
dy
Chap. 9 Sec. 9.2 E(X) = 1-p + 0-q = p Expectations of Sums and Independen! Products
257

for each i. Consequently,

E(S) = P + P + - - - + p = np
(Look back at the derivation in Section 4.6 to see how much easier this method is!)
Example 9

Let Sn be gamma distributed: Sn is the waiting time until the nth failure in a renewal process in which each component has an exponentially distributed lifetime with parameter X. In Section 6.6, E(Sn) was shown to be n/\ using an involved integration by parts. Let T be the lifetime of the z'th component. Since T is exponentially distributed with parameter X, E(T) = 1/X. But
Sn = T!
+ T.

implies that
E(Sn) =

E(Ttt) = -

1
X

n
X

with no difficult integration! Another random variable associated with a succession of Bernoulli triis is the number of runs R. For example, if a succession of 12 triis resulted in SSSFFSSFSFFF, then R = 6the number of distinct runs in either success or failure. To find E(R), the random variable R is decomposed as a sum. Let 1 O z'th result is different than (z - l)st z'th and (z - l)st results are the same

for z = 2, 3, . . . , n. For example, with 8 Bernoulli triis:

Sequence SSFFFSSS SSSFFFFF

R
3 2 1 6

fe 0 0 0

fe
1
0 0

fe
0

fe
0 0 0

fe
1
0 0 0

fe
0 0 0

fe
0 0 0

ssssssss
SFSSFFSF

0 0

Thus, to repeat, , is 1 if and only if there was a change between the (z - l)st and the z'th Bernoulli triis; that is, one of them was a success and the other was a failure. The sum
fe + fe + ' ' ' + i ,

is the number of times there was a switch from success to failure or from failure to success. This is one less than the total number of runs. That is,
R =
258

+ + & + 1
Chap. 9

Variances, Covariances, Correlation Coefficients, and More

(Check that this is true in the examples above.) Let us compute E(%) for any i = 2, 3, . . . . n: = 1) = P(ith result different than (z - l)st) = P(S on (/ - l)st trial and F on z'th) + P(F on (/ - l)st trial and S on z'th)
= pq + qp

Therefore, since the expectation of a sum is the sum of the expectations,

E(R) = fe) + + (U +

= 2pq + + 2pq + 1
= 2( - l)pq + 1

In a succession of n Bernoulli triis the expected number of runs is

E(R) = 2(n - l)pq + 1

Note that if p = 1/2 = q (fair coin), then
v

' 2 2

about half as many expected runs as triis. The expectation of a sum is the sum of the expectations. Is the same true for producs? Is the expeclation of a product equal lo the product of the expectations? Assume that X^, . . . , Xn are independen! random variables. expectation of the product is the product of the expectations: Then the

Xn) = E(Xl) E(Xn)

Let us emphasize the difference between sums and producs. It is always true that the expectation of a (finite) sum is the sum of the expectations, but only under the assumption of independence is it in general true that the expectation of a (finite) product is the product of the expectations. Still, the result has important applications. Proof. We will prove the result in the continuous case; the discrete case is similar. Suppose that X and Y have joint density function h(x, y). By independence h(x, v) is the product of the marginal densities / and g for X and Y,
Sec. 9.2 Expectations of Sums and Independent Products
259

Chap. 9

respectively. Thus
C C , .

E(XY)

=
J ce J co

xyh(x, y) dx dy

/*

xf(x) dx)yg(y)
J j

dy

E(X)yg(y) = E(X)E(Y)

dy

For producs of more than two random variables, E(X1---Xn) = ((*!- * = E(X1---Xn_1)E(Xn)

E(Xn}
The second equality uses the fact that the two random variables X1 Xn_l and Xn are independent combined with the result already proved for two random variables. Then the same technique is applied again and again to obtain the last equality. (Or use induction.)

PROBLEMS 9.14. The expected number of runs E(R) = 2(n l)pq + 1 in a succession of n Bernoulli triis. With the number n of triis fixed, graph E(R) as a function of p for O ^ p ^ 1. 9.15. Machine B makes bolts. Two percent are defective. To see whether a defective bolt tends to be followed by another defective bolt or whether they are independent, the results of 1000 bolts are recorded. Assuming independence from one bolt to the next (and so successive bolts constitute successive Bernoulli triis), what is the expected number of runs? 9.16. A shelf has 7 cans, 3 of which are dented. (a) You choose 2 without replacement. Let X be the number of dented cans in the sample. What is E(X)1 (b) Suppose that sampling was done with replacement. What is E(X) now? 9.17. (Continuation) Suppose that 3 cans are selected. What is E(X) when sampling is done (a) without replacement; (b) with replacement? 9.18. (The Hypergeometric Distribution: Continuation) A bin contains G green and R red blocks. N = G + R. n blocks are selected without replacement. Let X be the

260

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

number of green blocks in the sample. Let 1 O ith selection results in a green block tth selection results in a red block

(a) Show that X = ^ + + ^n. (b) What is Pfa = 1)? (See Section 3.3.) (c) What is E(X)1 (d) What is E(X) if sampling is performed with replacement?

Let X be the number of green blocks in a sample of size n from a bin of N blocks, G of which are green. For n ^ G,

regardless of whether sampling is performed with or without replacement.

9.19. Let X be the number of aces Stan receives in a game of bridge. (Each player receives 13 cards.) (a) Show that

P(X = j) =

y/U-/
52 13

4U 48

and iom last

(b) Find E(X) directly from the definition as

LoiP(x = i)
9.20. (Continuatior) Here is a tricky way to fnd E(X): Let X1, . . ., X4 be the number of aces that each of the four players receives. Reason that E(X) = E is the same for each i = 1, 2, 3, 4 and that E(X^ + X2 + X3 + X4) = 4 since Xl + X2 + X3 + X4 is the total number of aces received by all the players. Conclude that E(X) = 1. 9.21. Generalize the result about the expected number of runs: Suppose that n triis are performed; they are independen! and each can result in A with probability p, B with probability q, and C with probability r where p + q + r = 1. Find the expected number of runs.

mili
E7 SS

tive ;nt. the ted

9.3 THE VARIANCE

nt. ase
: is

ed he

The expectation is one number associated with a random variable. The variance is another; it is a measure of the variation of the random variable from the mean. Random variables X and Y with the density functions in Fig. 9.1 have the same mean \. But the vales of X are distributed more closely to E(X) than are the vales of Y to E(Y). That is, in any realization (in the result of a random experiment) the probability is greater that X will be closer to |x than Y will be. To fnd a single number that will be an indication of the possibility for variation from u,, consider the vales X - (JL and Y JJL. X - jx will tend to be cise to O, but

Sec. 9.3

The Variance

261

Density for X Density for Y

Figure 9.1

Y - \L will vary far from O either positively or negatively. To avoid cancellation, consider the squares: (X u.)2 will tend to be cise to O, but with greater probability (Y - u,)2 will be a large positive number. Consequently, E[(X - u.)2] will be less trian E[(Y - (x)2].
Defnition

Let X have mean |JL. The variance of X is <r2 = Var (X) = E((X - jji)2) The standard deviation is the square root of the variance: a = VVar (X) That is, u, is subtracted from each valu of X to obtain the new random variable X - |x. Then Var (X) is the expectation of the square.
Example 10
l

Recall that if X denotes the result of a die roll, then E(X) = 7/2. Find Var (X). Solution A probability table for (X - 7/2)2 is formed:

X
(X - 7/2)2
Probability

25/4

9/4 1/6

1/4 1/6

1/4 1/6

9/4 1/6

25/4

1/6

1/6

Thus
Var (X) = E( ( X - 25 4 _ ?5 ~ 12

25

Here is a program that simulates the variance of a die roll: First N die rolls are simulated. The result of the th is stored in DIE[I]. Then the average of all N
262

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

is computed as an estmate of JJL (although in fact we know that |x = 3.5). Then the average deviation is formed as an estmate of a2. - ^ (DIE[I] - AVERAGE)2 N 1=1
const NORM = 10000; ADDR = 4857; MULT = 8601; var SEED,I,N: nteger; RND,AVERAGE,VARIANCE: real; DIE: array [1..1000] of nteger; begin readln(N); SEED:= 0;
for l:= 1 to N do , begin SEED:= (MULT*SEED RND:= SEED/NORM;

ADDR) mod NORM;

DIE[I]:= trunc(6*RND + 1); end; AVERAGE: = 0.0; for l:= 1 to N do AVERAGE: = AVERAGE + DIE[I]; AVERAGE: = AVERAGE/N; VARIANCE:= 0.0; for l:= 1 to N do VARIANCE: = (DIE[I]-AVERAGE)*(DIE[I]-AVERAGE); VARIANCE:= (VARIANCE/N); writeln(AVERAGE:10:4,VARIANCE:10:4)

end.

Example 11

Let X be a Bernoulli random variable. Find Var (X). Solution X = 1 with probability p and O with probability q. Thus
IJL = E(X) = 1 p + O q = p

Construct the probability table for (X (ju)2:

X
(X -

0
P2

i
(i - p)2 p
263

|X)2

Probability

Sec. 9.3

The Variance

Therefore,
Var (X) = E((X

= p2q + (1 - p)2p = p2q + q2p = pq

Example 12 Let X be uniformly distributed on [a, b]. Find Var (X), Solution Recall that E(X) = (b + a)!2. Therefore,

Var (X) = E ( ( X x -

b + flN
b +a
1 dx a

1 1/ b +a r Ix - a 3
1 1 b - a 3

8
m

(b ~ a) 12

Finding Var (X) directly from the definition as was done in the last three examples can be complicated. Another formula is usually used instead. Noting that the expectation of a sum is the sum of the expectations, we see that

Var (X) = E((X -

= E(X2 = E(X2) =

|x2)

E(X2)

2|X (JL +

|JL2

Or, finally,

Var (X) = E(X2) - v2

This formula provides the computationally easiest method for finding variances. Example 13 Let X be exponentially distributed with parameter X. Find Var (X). Solution By Example 4 E(X") = n!/X". Consequently,

Var (X) = E(X2) - |x2 =

264

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

Example 14 Compute the variance of W where W is geometrically distributed. From Section 4.6 E(W) = /p. To find E(W2) involves a similar technique of interchanging a derivative and a sum. In fact, E(W(W 1)) is easier to find. So first this quantity is computed and then E(W) is added to it to find E(W2). To find E(W(W - 1)), note that if W takes on the valu /, then W(W - 1) takes on the valu j(j - 1).

- i)) =
7=1 7=1

;(; - i)<

d2 '*dq2
7

dq2

-q

2pq (1 -

2q
Thus
E(W2) = E(W2 - W + W)
2

- W)

- 1)) + E(W)

Finally, manees.

= E(W2) - j,2
2q
= +

7 P~ O! 2q + p - 1
P2

Sec. 9.3

The Variance

265

Given a random variable X, Var (X) is the expectation of the positive random variable (X - (j,)2. Since (X fji)2 cannot take on negative vales, the farther this random variable is from O, the larger will be its expectation. That is, the farther X can be from its expectation u, with large probability, the larger will be the variance of X. In fact, multiplying X by a factor of 2 will result in multiplying the variance by a factor of 4. More generally, let us see what the effect of a change of scale is on the variance: Let a and b be constants. Then
Var (aX + b ) = a2 Var (X)

= \a\aj. Of course, the result about the standard deviation is simply the square root of the first equality. Before proving the first equality, note that the result is intuitive; the vales of the random variable aX + b vary from the expectation E(aX + b), as do the vales of aX from its expectation E(aX). Note also the special case when a = 0: The variance of a constant is 0:
Var (b) = O

In gambling terminology: Suppose that you arepaid $b for playing the game. Then your winnings will be $b with O variance. To prove the change-of-scale formula, let (x = E(X). Note that

E(aX + b) = aE(X) + b = a\i + b

Thus
Var (aX + b) = E((aX + b)2) - (E(aX + b))2

= E(a2X2 + labX + b2) - (au. + b}2 = a2E(X2) + 2abE(X) + b2 - (V + = a2(E(X2) - u2) = a2 Var(JT) Computations of the variance can get fairly involved. But there is a basic result that can be applied to sums of independent random variables which simplifies difficult sums and integris. Recall how in Example 8 the mean of a binomially distributed random variable was found by decomposing it as a sum of other random variables; and later, the same technique was used to find the expected number of runs in n Bernoulli triis. In fact, a similar technique can some times be used to compute variances.

+ b2)

266

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

The variance of an independen! sum is the sum of the variances: If XJ, . . . , Xn are independent, then

Var (Xl + + Xa) = Var (X

+ Var

Proof. The result is proved for two random variables X and Y. The proof for more than two is similar. Assume that X and Y are independent. Thus E(XY) = |x^|xy, where u.^ and |xy are the expectations of X and Y, respectively. Then

Var (X + Y) = E((X + Y)2) - (^ + M2

= E(X2 + 2XY + Y2) - (& + 2^X^Y = E(X2) - , + (72) - & + 2E(XY) after a reordering of the terms to produce the last line. Since the expectation of XY is the product of the expectations as noted above, the last two terms cancel. The first two terms are Var (X), the next two terms are Var (Y). Exatnple 15 Suppose that S is binomially distributed n, p. Decompose S as the sum of n Bernoulli variables as in Example 8:

S = X1 +

where X is 1 or O if the zth trial resulted in success or failure, respectively. Since X-L, . . . , Xn are independent,

Var (S) = Var (XJ + + Var (Xn)

But Var (X,) = pq by Example 11. Therefore,

Var (5) = pq + + pq = npq

Example 16

Suppose that X is Poisson distributed with parameter X. Problem 9.29 asks you to compute a series to show that Var (X) = \. Thus the variance and the expectation of a Poisson distributed random variable are both equal to the parameter \. But here is a way to see this fact using Example 15. Suppose that 5 is binomially distributed, n triis, p = probability of success. Then the distribution of 5 approximates the distribution of X if n large p =O X = np

But Var (S) = npq = (np)q = \q = X since p cise to O implies that q is cise to 1. Henee we should not be surprised to find that the formal proof in Problem 9.29, shows that the variance of X is X.

Sec. 9.3

The Variance

267

If X is binomially distributed, n = number of triis, then

Var (X) = npq

If Y is Poisson distributed with parameter X, then

Var (7) = X

E(X) determines where Xis "centered" and Var (X) measures how the vales of X are spread about that center. A change of scale changes both of these. Let |x be the expectation of X and a2 be the variance. variable Z obtained by a change of scale
Z =
X-

Then the random

is standardized. That is,

E(Z} = O
Var (Z) = 1

To see why this is so, we apply the results concerning changes of scale:
E(Z) = E~Var (Z) =

(E(X) - MO = O

-X- o a

= 2 Var (X) = 7 a2 = 1 cr -

Example 17 Let X be uniformly distributed on [a, b]. Let Z be the standardized versin of X. That is, Z = (X |x)/cr, where |x and o- are the mean and standard deviation of X. Find the density of Z. Solution Example 12 shows that
ix = E(X) =

a + b

a = VVar (X) = Thus

Z =

b -a VI2

X - (a + b)/2 (b - a)/VT2

When X = a, Z = -VT2/2 = -V5. And when X = b, Z = + V/2 =

268

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

V3. Thus Z is concentrated on the interval [ - V3, + V3]. For z in this interval,
z)

..

(b - o)/Vl2
b - a
1 b -a

a +b a +b

Zb

vales se.

^n + ~a
- a
~*~ Figure 9.2 P(Z =s z) is the fraction of the total length taken by the hashed segment.

iom

Note that this last expression has been obtained by the fact that the probability that X resides in an interval is proportional to the fraction of the entire interval [a, b] occupied by the subinterval. (see Fig. 9.2)
b - a a +b

Taking derivatives implies that the density of Z is

rO
1 z < - V3

or + V3 < z

V12

-V3 < z < +V3

Note that not only is Z uniformly distributed (constant density on the interval), but Z does not depend on the endpoints a and b defining the density of the original random variable X. Thus every uniformly distributed random variable, when standardized, has the density of Z. PROBLEMS

ion of 'iation

i 9.22. For the random variable in Problem 9.1, find (a) Var (X); (b) Var (X2); (c) Var (ex). \ 9.23. (Continuation) Find the density of the standardized versin of the random variable X of Problem 9.22. That is, find the density of (X - |x)/a. 9.24. Let X be exponentially distributed with parameter X. Find the density of the standardized versin of X. Show that it does not depend on the parameter X. 9.25. (a) For the random variables F and C of Problem 9.5, find Var (C). (b) Also find Var (F) using the density function of F. (c) Verify that Var (F) = (9/5)2 Var (C). : 9.26. Let Y = 3X - 4, where Xis as in Problem 9.1. Find the density function for Y and use it to find (a) E(Y); (b) Var (Y). Verify that (c) E(Y) = 3E(X) -- 4; (d) Var (Y) = 9 Var (X). 9.27. Suppose that W is the waiting time until the first success in a series of Bernoulli triis or N + 1, whichever comes first (see Problem 4.38). Find Var (W) and show that Var (W) -> q/p2 as N - . 9.28. For any random variable X, show that Var (X) = E(X(X - 1)) + E(X) Chap. 9 Problems

2/2 =

(E(X))2
269

hap. 9

9.29. Let X be Poisson distributed with parameter X. Evalate

E(X(X - 1)) = /(/ - 1) * e-^

Use the result of Problem 9.28 to conclude that Var (X) = X. 9.30. Let X be a random variable and let t be a fixed real number. Define the function g by

g(t) = E((X - O2)

Thus when t = \L, g(t) = Var (X). Show that

= Var (X) + (IL - t)2

Conclude that g is minimized for t = \. Thus the smallest valu of E((X )2) is Var (X), which occurs when t = |ju THE COVARIANCE AND CRRELA TION COEFFICIENT

In seeking a few numbers that summarize how a random variable is distributed, we developed the expectation and the variance. For two random variables X and Y, is there a number that summarizes their relationship? At one extreme X and Y are independen! so that knowledge of the valu of X indcales nothing about the valu of Y; at the other extreme, Y = X. Suppose that large vales of X tend to be associated with large vales of Y. For example, let X be the tempera ture and Y equal the number of molecular reactions between chemicals for which heat acts as a catalyst. More specifically, suppose that whenever X is larger than its mean, u,^-, Y is also larger than (juy; and conversely, whenever X is smaller than u,^, Y is also smaller than \*.Y. Then the product (X -- \x)(Y - \Y) is always nonnegative since both factors are either positive or negative. Thus the expectation of the product will be positive. On the other hand, if vales of X larger than u,^ tend to be associated with vales of Y smaller than u,y, the product (X - |x^)(Y -- |xy) will tend to be negative and will therefore have a negative expectation. Defnition Suppose X and Y are two random variables with expectations The covariance between X and Y is and JJLK.

Cov (X, Y) = E((X -

Example 18 Let X be the number of heads in three flips of a fair coin and Y be the number of runs. Find Cov (X, Y).
270

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

Solution Let us construct a probability table with first row the sample points:
Outcome
X Y
Probability

TTT 0 1 1/8

TTH 1 2 1/8

THT 1 3 1/8

HTT 1

HHT 2 2 1/8

HTH 2 3 1/8

THH 2 2 1/8

HHH 3 1 1/8

2 1/8

fe can be calculated without reference to the table since X is binomially distributed n = 3, p = 1/2, so E(X) = np = 3/2. Also,
1 + 2 + 3+ 2+ 2+ 3+ 2+ 1

=2

[Note that Y is the number of runs in n = 3 Bernoulli triis with/? = 1/2; therefore, from Section 9.2, E(Y) = 2(n - l)pq + 1 = 2.] Thus Cov(X,Y) = El\X--](Y-2)

-2<>
(0)
= O
+

(1) * ( O

Note that if X = Y, then

Cov (X, X) = E((X -

= Var (X)

by definition. Thus the covariance is actually a generalizaran of the variance to more than one random variable. And just as the variance is computed using the fact that it is the expectation of the square minus the square of the expectation, the covariance has an expression which allows for easier computation:
Cov (X, Y) = E(XY)

In words, the covariance is the expectation of the product minus the product of the expectations. To see this expand the definition:
Cov (X, Y) = E((X - fe)(Y - M,y)) = E(XY - fe Y - [YX +

= E(XY) - fe(Y)
Sec. 9.4 The Covariance and Correlation Coefficient 271

= E(XY) - \L
=

E(XY)

Example 19 (Continuation of Example 6) Two cards are selected. X = number of hearts, Y = number of spades. Example 6 we showed that E(XY) = .1275. Also from the table, E(X) = 1(.3824) + 2(.0588) = .5 E(Y) = 1(.3824) + 2(.0588) = .5 after computing the marginis. Thus = .1275 - .25 = -.1225 Cov (X, Y) = E(XY) - E(X)E(Y) In

The covariance is negative because there is a negative association between X and Y: The presence of one suit tends to decrease the likelihood of cards of the other suits. Suppose that X and y are independent random variables. Then in Section 9.2 we showed that the expectation of XY is the product of the expectations , that is, that E(XY) = feM-y- Thus If X and y are independent, then
Cov (X, y) = O

The converse ofthis result is false. The random variables X and Y of Example 18 nave covariance O, but they are not independent. For the two vales X = 2 and y = 2, P(X = 2 and Y = 2) = P(HHT or THH) = ^ P(X = 2)P(Y = 2) = P(two heads)P(two runs)
3 4

Thus covariance O does not imply independence.

Still, Cov (X, y) can be used as a measure of how nonindependent X and Y are. Cov (X, Y) = O if X and Y are independent. So the larger Cov (X, Y) is, either positively or negatively, the less independent X and y will be. On the other hand, if the vales of X and/or Y happen to be large, even though X and Y might be "nearly" independent, the covariance might be large simply because X and/or y take on large vales. To remedy this situation and define a quantity that truly
272
Variances, Covariances, Correlation Coefficients, and More Chap. 9

measures how closely X and Y are associated, first X and Y are standardized and then the covariance is taken. Before going into the details and examples, let's see how a change of scale affects the covariance. Let a, b, c, d be constants. Then

Cov(aX + b,cY + d) = ac Cov (X, Y) Note that the special case X = Y, a = c, and b = d is the od result about the effect of a change of scale on the variance: Cov(aX + b, aX + b) = Var (aX + b)

= a2 Var (X) = a2 Cov (X, X) To prove the result in general involves using the basic formula for computing covariances: Cov(aX + b,cY + d)
= E((aX + b)(cY + d)) - E(aX + b)E(cY = E(acXY + adX + bcY + bd) - (a\ix + = acE(XY) + ad\Lx + >C(xy + bd (a = ac(E(XY) - u,^y)
d)

+ d)

= ac Cov (X, Y) Now we are ready to define a measure of association between X and Y. We start with the derivation and then summarize in a formula that can be used for computations. Let X and Y be random variables and let
X -

Z, =

yCTy

be the standardizations of X and Y. That is, Zt and Z2 are obtained from X and Y by changes of scale so that Z1; Z2 have O expectations and variances 1. The correlation coefficient between X and Y is defined to be Cov (Zls Z2). But it is almost never computed using this definition. In fac,
Cov (Z1; Z2) = Cov
X -

\X

Y -

Cov (X, Y)
See. 9.4 The Covariance and Correlation Coefficient
273

by the change-of-scale formula for the covariance: a = l/ax, b = \y.xlax, c = 1/oy, and d = |xy/cry. So we arrive at a new definition equivalent to the one abo ve. Definition The correlation coefficient between X and F is

i, F) = ^V (*' F)

Example 20 Find p(X, Y) where X and Y are as in Example 19. Solution By that example Cov (X, F) = -. 1225. The variances are needed; the marginal distribution of X is (from Example 6)
x
Probability .5588 .3824 .0588

Therefore, E(X2) = 02(.5588) + 12(.3824) + 22(.0588)

= .6176
Since E(X) = .5 (either from the table or Example 19), Var (X) = .6176 - (.5)2 = .3676 By symmetry in the original joint probability table, the marginal distribution of F is the same as the distribution of X. Thus Var (F) = Var (X). Finally,
P(X,

Cov (X, F)

-.1225

= -.3332

Using the correlation coefficient as the measure of association between X and F has a very convenient property: It is unchanged as the result of a change in scale. For example, if X is the temperature at noon on Saturday and F is the temperature at noon on Sunday, then p(X, Y) is the same whether Celsius or Fahrenheit temperatures are used. Or suppose that X records the chickens crossing the road from right to left and F records the chickens crossing from left to right. Then p(X, F) is the same whether the actual numbers of chickens are measured or whether the number of tons of chickens is measured (assuming that the chickens crossing in either direction weigh the same).

274

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

c =

The correlation coefficient is invariant under changes in scale: Let a, b, c, d be constants. Then

p(aX + b, cY + d) = p(X, Y)
where the sign is + if a and c have the same sign if a and c have opposite signs

The technicalities about the sign are a result of the calculations in the proof below. But note that if one change of scale is applied to both X and Y, then a = c, b = d and

p(aX + b, aY + b) = + p(X, Y)
Proof. Use the definition of p in trras of Cov and the effect of changes of scale on the covariance:

Cov (aX + b,cY + d) p(aX + b,cY + d) = -----'ac Cov (X, Y) \a\(Tx\c\aY Cov (X, Y)

which completes the proof since this is p(X, Y) by definition.

We showed that if Xand Fare independent random variables, then Cov (X, Y) = O and consequently p(X, Y) = 0. Thus the correlation coefficient is a measure of the degree of nonindependence. In fact, much more can be said.
Result

For any two random variables X and Y,

-1 P(X, Y) =s +1

If p(X, Y) = 1, one of the random variables is a linear function of the other. Thus we see that a large valu of the correlation coefficient implies a direct relationship between X and Y. Proof of the Result. We can assume that X and Y are standardized with

E(X) = O = E(Y).

Var (X) = 1 = Var (Y)

See. 9.4

The Covariance and Correlation Coefficient

275

This is so since if X and Y are not standardized, then replace them with their standardizations X' and Y'; we know that the correlation coefficient will remain the same:
p(X', Y') = p(X, Y)

Since X and Y are standardized, we also have these relationships: E(X2) = E(X2) - (E(X))2
2 2 2

= Var (X) = 1

(Y ) = (Y ) - (E(Y)) = Var (Y) = 1

Cov (X, Y) = E(XY) - E(X)E(Y)

Cov (X, Y)
crxcrY

= E(XY)

= E(XY)

But the random variable

Z = (X - Y)2

is nonnegative. Consequently, its expectation is nonnegative. In fact, since all the vales of Z are greater than or equal to zero, E(Z) 5* O and E(Z) = O implies that Z = O and thus X = Y. That is,
E((X - Y)2) s* O

and
E((X - Y)2) = O

impies that X = Y. However,

O E((X - Y)2)

= E(X2) - 2E(XY) + E(Y2)

= 1 - 2p(X, Y) + 1 = 2(1 - p(X, Y))

Thus

P(X, Y) 1
with equality holding if and only if X = Y. By considering the random variable Zj = X + Y rather than Z = X Y, similar reasoning implies that

with equality holding if and only if X = Y. Thus always

, Y) *s + 1
If p(X, Y) = +1, then X = Y; if p(X, Y) = - 1, then X = - Y. Now suppose that X and Y are not necessarily standardized. By what we have just shown, if
276

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

i with their will remain

X' denotes the standardization for X and Y' the standardization for Y, then

But
x, =

X CTj

Y -

Andif p(A", Y') = 1, then

X' = Y'
CTy CTy

Solving this last equation for X shows that X is a linear function of Y. In fact,

^ = \ v _L X F+
OY

fe + |Xy

:t, since all = O implies

What does the valu of the correlation coefficient p(X, Y) allow us to conclude? If p(X, Y) is 1, then one is a linear function of the other. If p(X, Y) is cise to 1, then X is "cise" to being a linear function of Y. For example, perhapsXis the number of people enterng a theater and Fis the number of tickets sold. Then on any one day X is nearly Y: All people entering the theater buy tickets except for a few who are picking up kids or who are using the phone, and so on. How cise to 1 is cise enough. Vales of .9 are considered to be highly significant in the more exact sciences. Vales of .7 are considered significant in the social sciences. Now suppose that the valu of p(X, F) = 0. It is tempting to leap to the conclusin that X and F must be independent. But recall Example 18: There Cov (X, F) = 0; thus p(X, F) = 0. But X and F are not independent. If X and F are independent, then p(X, F) = 0. But if p(X, F) = O, then no conclusin as to the independence of X and F can be drawn. In fact, F might be a function of X and still p(X, F) = O (see Problem 9.38). Here is an example which shows how partially correlated random variables (O < p < 1) can arise from independent random variables: Example 21 Suppose that U, V, and W are independent random variables each with mean jx = 3 and variance or2 = 5. Let

m variable

wsuppose shown, if
Chap. 9

X = U+V
Find p(X, F).
Sec. 9.4

Y = U +W

The Covariance and Correlation Coefficient

277

Solution

E(X) = E(U + V) = E(U) + E(V) = 6 E(Y) = E(U + W) = E(U) + E(W) = 6

E(XY) = E[(U + V)(U + W)]
= E(U2) + E(U)E(W) + E(V)E(W)

u
E(V)E(W)

= E(U2) + 3-3 + 3-3 + 3-3

= E(U2) + 27
where the second equality uses the fact that the expectation of a product of independen! random variables is the product of the expectations. E(U2) = [(E(U2) - (E(Um + (E(U)Y = Var (U) + (E(U))2

= 5 +9 = 14
Thus Cov (X, Y) = E(XY) E(X)E(Y)

= (14 + 27) - 6 6 = 5
Also, since U, V, and W are independent,

Var (X) = Var (U) + Var (V) = 5 + 5 = 10 Var (7) = Var (U) + Var (W) = 5 + 5 = 10
Therefore,

P(X, Y) =

Cov (X, Y)

VioVTo

Thus random variables with correlation coefficients in ( 1, 1) often arise when they are "combinations" (sums, products, . . .) of partially overlapping sets of independent random variables. For example, the number of tickets sold at the theater on Monday "partially correlates" (has a valu of |p| between O and 1) with the number sold on Tuesday because some of the variables determining each are the same (season of the year, same movie on both days, . . .) and some are independent (weather, different groups of people on the two days, . . .).

278

Variances, Covariances, Correlation Coefficients, and More

Chap. 9

PROBLEMS In Problems 9.31 to 9.35, find E(X), E(Y), Var (X), Var (y), Cov (X, Y), and p(X, Y) for the random variables X and Y in 9.31. Example 5. J9.32. Problem 9.8. 9.33. Problem 9.9. , 9.34. Problem 9.6. I 9.35. Problem 9.7. Let X = A = number of apples and Y = B = number of bananas, -f 19.36. Suppose that U, V, and W are independent, each with mean u. and variance o-2. Let J X = U + V and Y = U + W. Find p(X, Y) in terms of p, and a. 9.37. (Continuation) With U and V as in Problem 9.36, suppose that X = aU + bV and y = ai/ + fiV, where a, 6, a, p are constants. Find p(X, Y) in terms of a, b, a, P, (Ji, and cr. 9.38. Let ^T be uniformly distributed on [-1, 1]. Let y = Jf 2 . Show that p(j$f, y) = O even though X and y are as far from being independent as two random variables can get: Given the valu of X, Y = X2 is completely determined! 9.39. (A Time-Series Model) Each day an amount of pollutant X is dumped into the river. This amount decays (dissipates and degenerates) at an exponential rate so that days after dumping there will be an amount ot'X still active, where O < a < 1. On day t assume that X, was dumped where
y y -y -y y

'

' '

7 -***- 2' J^ 1'

O'

15 ~"2' * * '

are IIDRVs. Snce they are identically distributed, they have a common mean (j, and variance a2. Show that the amount of pollutant still active in the river on day t is S, = 2 a'Xt- Show that
E(St) =
(1 - a)2
1 - a2

Var (S,) = For t ^ s,

a2 1 - a2

E(S, Ss) =
Cov (5,, 5.) =

(1 - a)2
1 - a2

1 - a2

p(S,, S,) = a'^

(In your computations assume that the expectation of the infinite sums are the sums

Chap. 9

Problems

279

of the expectations, and similarly the variance of the infinite sums of independen! random variables are the sums of the variances.) 9.40. Show that

Var (X + Y) = Var (X) + 2 Cov (X, Y) + Var (Y)

for any random variables X and Y. 9.41. (Continuation) Show that

Var (X + Y + Z) = Var (X) + Var (y) + Var (Z) + 2 Cov (X, Y) + 2 Cov (X, Z) + 2 Cov (y, Z)

9.5 SUMMAfY OFE, VAR, COV, AND p PROPERTIES Let a, b, c, d be constants. Then

1. 2. 3. 4.

E(c) = c. E(aX + b) = aE(X) + b. E(X, + + Xn) = E(Xl) + . - + E(Xn). If X^ . . . , Xn are independen!, then E(X1
n)

= E(X1) E(Xn)

5. 6. 1. 8.

Var (c) = 0. Var (aX + b) = a2 Var (X). Var (X) = E(X2) - (E(XJ)Z. If X-i, . . . , Xn are independent, then Var (JTj + " + *) = Var + Var (*)

9. If X has expectation jx and variance o-2, then

A - - M.
^

is the standardized versin of X:

E(Z) = O
10. 11. 12. 13.

Var (Z) = 1

Cov pT, X) = Var Cov (X, Y) = (XY) - E(X)E(Y). Cov (0^ + b), (cY + d)) = ac Cov (X, Y). If X and y are independent, then Cov (X, Y) = O and

p(JT, F) = O

14. -1
280

, y) ^ +1.
Variances, Covariances, Correlation Coefficients, and More Chap. 9

independent

15. p(aX + b, cY + d) = p(X, Y). 16. If p(X, Y) = 1, one is a linear function of the other.
DISCRETE RANDOM VARIABLES

(Y,Z)
0

with prob q with prob p

CONTINUOUS RANDOM VARIABLES Random variable

Density Expectation Variance

hap. 9
Sec

- 9-5

Summary of E, Var, COV,

and p Properties

281

Oft expectation fails, and most oft there Where most it promises William Shakespeare, 1564-1610, All's Well that Ends Well

Chapter 10 The Normal Distribution, Central Limit Theorem, and Law of Large Numbers
The normal distribution with its associated "bell-shaped" density function is one of the most useful distributions; it forms the basis of much of statistical analysis. The reason for this is the central limit theorem, which states that many distributions can be approximated by the normal distribution. The theorem was first proved by Abraham de Moivre (1667-1754) and extended to the general binomial distribution by Pierre-Simon de Laplace (1749-1827). In many ways, de Moivre and Laplace were opposites; de Moivre was unpretentious, of modest means, unable to find a university position, living on his earnings as a tutor; Laplace was well established, a monarchist during the time of the French revolution. Laplace was the foremost contributor to probability theory during its early years.
10.1 THE NORMAL DISTRIBUTION

Almost simultaneously Cari Friedrich Gauss (1777-1855) and the American Robert Adrain (1775-1843) in the early nineteenth century developed a "theory of errors." Suppose a quantity is measured several times; each observation has a slight error due to inexactness in measurement. How are the errors distributed? They will be random, but what is the specific distribution function? Both Gauss and Adrain were led to the normal distribution, a variant of which is still called the error function. Suppose that the random variable 'is due to very many small contributions; X might be the temperature or the total winnings after many gambling games or
282

a student's exam score or the number of people in the bus depot or. saw in Chapter 9, the random variable
Z =
X-

. As we

cr

obtained by subtracting the mean of X and dividing by the standard deviation is standardized: E(Z) = O, Var (Z) = 1. The "law of errors" states that Z is normally distributed: The normal distribution mean O, variance 1 (abbreviated as N0 ^ has density function

i
and distribution function
e "2/2 du
2-TT J-

Example 1 Graph 4>. Solution 4> is a function of x2; henee the graph of 4> is symmetric about the y axis.

Therefore, the slope of the tangent to the graph of c|> is O for x = 0. points (4>"(*) = 0) occur at x = 1. (see Fig. 10.1)

Inflection

To check that 4> is in fact a density function requires showing that the total rea under the graph of (J> is 1. This requires a tricky maneuver with polar co-

Figure 10.1

Sec. 10.1

The Normal Distribution

283

ordinates due to Laplace. Let / = e-*2/2 dx

We need to show that / = A/ZTT or, equivalently, that /2 = 2ir.

I2 = I I

r
J 0=

e~x2/2 dx
J

e-y2/2 dy

dx

dy
dy

-Li = 11
fnr

/*)'

-x2/2

-y

The second equality is true because whether we use the dummy variable x or y as the variable of integration is immaterial to the actual valu. The third equality follows from interpreting x and y as coordinates in two-dimensional xy space; since 2 x2/2 and the integral sign for e-* /2 js a constant with respect to the variable y, e~ the y variable can be interchanged. The fourth equality holds since e~y2'2 is a constant with respect to x, so e~y2'2 and dx can be interchanged. Switch to polar coordinates; note that
x = r eos 6 y = r sin 9

and the infinitesimal rea element is

dx dy = r dr dB

As x and y both range from oo to +00, the entire plae is spanned; thus r ranges from O to oo and 6 ranges from O to 2ir.
I2 =

r dr i

-r
Jo
- 012^

ide

- lo

= 2-ir

This completes the demonstration that I2 = 2ir and henee that / = <|> is normalized and therefore is a density function, as claimed.
284
The Normal Distribution, Central Limit Theorem

Thus

Chap. 10

To show that cj> has mean O and variance 1 requires evaluating these integris:

E(Z) = I
J
-00

x$(x) dx = 1= J

xe~x212 dx = O x2e~x2/2 dx - O = 1

Var (Z) = E(Z2) - E(Z)2 =

These two form the content of Problem 10.8. Unfortunately, there is no technique of integration that will allow an indefinite integral of $ to be found in terms of a simple expression involving "ordinary" functions. That is, it is difficult to show, but nonetheless true that there does not exist a function F which is a relatively straightforward combination of familiar functions (polynomials, trig functions, exponentials, logarithms, . . .) such that F' = c|>. There is a function <E> which is the indefinite integral of 4>, but it is a new function. The integral of $ from - to x is denoted by $(*). The error function is defined to be 2<J>(V2~) - 1. Vales of $(*) must therefore be found using approximations to the integral. They are tabulated for vales of x from 0.00 to 3.49 in Appendix 3. For x ^ 3.50, $(*) is nearly 1. Example 2

Let Z be a random variable N01 distributed. Find (a) P(Z > 2.3), (b) P(Z < -1.7), (c)P(-.8<Z< .8), ad(d)P(-.3 < Z < .7)
Solution (a)

P(Z > 2.3) = 1 - P(Z 2.3)

= 1 - $(2.3) = 1 - .9893

= .0107

(see Fig. 10.2) (b) By symmetry, P(Z< -1.7) = P(Z> 1.7)

= 1 - <>(1.7) = 1 - .9554 = .0446

rea = 1 - 4>(2.3)

2.3

Figure 10.2 Note exaggeration of vertical axis.

285

Sec. 10.1

The Normal Distribution

*fc

Area = 1 -

-1.7

1.7

Figure 10.3

(see Fig. 10.3) (c) Also by symmetry,

P(-.8< Z < .8) = P(X< .8) - P(Z -.8)
= $(.8) - P(Z > .8)

= 0>(.8) - [1 - P(Z s .8)] = 2 d>(.8) - 1

= 2 .7881 - 1

= .5762

(see Fig. 10.4)

rea = *(0.8) -4>(-0.8)

-0.8

0.8

Figure 10.4

rea B' = rea B

Figure 10.5

286

The Normal Distribution, Central mit Theorem

Chap. 10

(d) Finally,

P(-.3 < Z < .7) = P(Z < .7) - P(Z *

= <&(.?) - [1 - <D(.3)] = .7580 - [1 - .6179] = .3759

-.3)

Note: To find prbabilities involving the N01 distributed random variable Z, it is easier to remember how to draw a quick sketch and use it to find the required probability than it is to remember various formulas. For example, i f s , t s ? Q , then P(-s < Z < i) = rea A = <J>() - rea B =$() - (1 - 3>(s)) = 3>(f) + 4>() - 1 as in Fig. 10.5

10.2 PROPERTIES OF NORMALLY DISTRIBUTED RANDOM VARIABLES

In general, a normally distributed random variable need not have mean O and variance 1. For example, if Z is 7V0jl distributed, then A" = aZ + b obtained by a change of scale will still be normally distributed although E(X) = b and Var (X) = a2.
Definition

Suppose that X has mean u. and variance cr2. X is normally distributed x, cr2 (abbreviated N^) if the standardized random variable
Z =
is yV0 j distributed. Example 3
X (JL

Past experence has shown that the number of cars entering a park on a summer's day is normally distributed with mean 200, variance 900. With Y equal to the number of cars, find P(X ^ 220)
Solution

P(X > 195)

P(190 < X < 210)

Z =

X - 200 30

is N01 distributed. So we express the various events in terms of Z and then use the table.
m Chap. 10 Sec. 10.2 Properties of Normally Distributed Random Variables
287

P(X * 220) = p(*-=. ^) = P(Z * .6667)* = .7486

30 30 30

= P(Z> -.1667) = .5675

10
<Z<

10 (~ 30~

30

= P(-.3333 < Z< .3333) = 2 .6293 - 1 = .2586

In finding a probability involving the N^ distributed random variable X, the general technique is to express the probability in terms of the N0 distributed variable Z = (X - u,)/cr and use the table. Example 4 Experience has shown that a certain small company uses its phones for a random time Jeach month where Tis normally distributed with mean 300 hours, variance 80. Find an interval centered around 300 hours so that we can be 90% sure that the time Tfor the next month will fall in this interval. Solution A constant c is sought which satisfies the equation .90 = P(300 - c < T < 300 + c)
T - 300 < V80

Vvi <
= p
From the table and Fig. 10.6

-c

80

V8/

V80

= 1.645

c = 14.71
Conclusin: Ninety percent of the time (9 out of 10 months) the total phone usage Y will satisfy 300 - 14.71 < T < 300 + 14.71 or T e (285.3, 314.7) Let us find the density function for an N^ random variable X. Since Z =
*For simplicity, interpolation is not used in the table. For example, to find P(Z =s z), z is simply rounded to the nearest one-one hundredth.

288

The Normal Distribution, Central Limit Theorem

Chap. 10

rea = 0.90 thus rea to left of C/V80S0.95

Figure 10.6

(X - |x)/cr is N01, the distribution function for X is

Fx(x) = P(X ^ x)
(X - p x -

since Z has distribution function $. Differentiate using the chain rule:

(x a \ 1

cr

exp exp

Lt

a
2a2

'2-rcr

Alternative definition of a normal mean JJL, variance u2 random variable: is N^ ff2 distributed if its density function is
TTCT

2cr2

However, this formula is of limited usefulness since when dealing with an N^ random variable X, it is almost always simples! to express all computations in terms of the N01 distributed random variable Z = (X - (Ji)/a obtained by standardizing X. Note that the formula does in fact become the TV^ density function when (JL is taken to be O and cr2 is taken to be 1. Fig. 10.7 shows normal density curves with various vales of JJL and cr. A distinguishing feature of the normal distribution is that independen! sums of normally distributed random variables are again normally distributed. (Recall
Sec. 10.2 Properties of Normally Distributed Random Variables
289

-2

-1

Figure 10.7

from Chapter 8 that this property is also true of Poisson distributed random variables.) Let Xi, . . ., Xn be independent; suppose that X is normally distributed JJL,-, of. Then
+

is normally distributed with mean m + + in and variance cr^ +

The mean of the sum of the random variables is the sum of the means; the variance of the sum (not the standard deviation) is the sum of the variances. The proof involves intricate computations with integris. First the result is proved for two normally distributed random variables X and Y; then the result is extended by induction to more (see Problems 10.10 and 10.11). Thus the bell290
The Normal Distribution, Central Limit Theorem Chap. 10

shaped normal density curve "propgales ilself" in sums of variables each having Ihe bell-shaped normal densily curve. Example 5 Suppose the lime unlil a conslruclion crew compleles a building is normally disIributed wilh mean 90 days and slandard devialion 10 days. Afler completion, il lakes an addilional lime lo inslall Ihe plumbing, eleclricily, and finish Ihe inlerior. Assume Ihal Ihe addilional lime is independen! of Ihe building complelion lime and is normally dislributed wilh mean 30 days and slandard devialion 5 days. Le T be Ihe lime from Ihe beginning unlil Ihe inlerior is finished. Find P(T ^ 130 days). Solution T is normally dislribuled wilh

E(T) = 90 + 30 = 120
Var (T) = 102 + 52 = 125
Henee

Z =
is NQ dislribuled. Thus

T - 120 VI25
130 - 120 VT25

p( r =s 130) = p (z
= .8133

= P(Z = .8944)

var-

Generalizing from Example 5: A random experimenl is performed n limes and Ihe same quanlily is measured each lime. Cali Ihe result of Ihe Ih measuremenl X. For examples, X could be Ihe lemperalure al noon on Ihe th day of Ihe week or X could be Ihe heighl of Ihe ith person. Note thal X:, . . . Xn are IIDRVs. The colleclion {X,}"=1 is called a random sample. Since Ihey are idenlically dislributed, Ihey have a common mean and slandard devialion

o-2 = Var
The sample mean is Ihe average

x = -n (x, + + xn)
Nole Ihal

E(X) = - E(X,
Var (X) = \ Var (X^ + + Xn) = ^ n Var (X,) =
Sec. 10.2 Properties of Normally Dislributed Random Variables
291

The sample mean X has mean and variance

E(X) =

Var (X) =

In words, the sample mean has expectation equal to the population mean jx, but the variance is decreased by a factor equal to the sample size n. This confirms intuition: If a quantity is to be measured, it is better to take several measurements and use the average. Why? Because, as stated above, the average will have a smaller variance than the individual measurements. This property is very important; it explains why the sample mean is used widely in statistical estimation. Now suppose that each X is_normally distributed with mean JJL, and standard deviation a. Then the mean of X is also |x,_but Var (X) = a2/n. But the sum X1 + + Xn is normal distributed; thus X, which is the sum divided by n, is also normally distributed. Let {X,}"^1 be a random sample, each N^a2 distributed. Then the sample mean X is N.. !< distributed. The normal distribution reappears as the distribution of the sample mean if each X is normally distributed.
Example 6

Assume that the temperature for a random day in July is normally distributed with average 77 and standard deviation 9. Let X be the average temperature for a week. Find P(72 = ~X 80). Solution Under the assumption of independence X is N771/7 distributed. Thus

P(72 as X *z 80) = P

72-77

X - 77 80 - 77

PROBLEMS 10.1. Z is NQ1 distributed. Find (a) P(Z < .6); (b) P(Z =s - .03); (c) P(.12 < Z = 2.14); (d)P(-1.67 Z < 1.20). 10.2. XisN_w16 distributed. Find (a) P(X > -12); (b) P ( - l l < X ^ -8);(c)P(|Z + 10| < 2);\A)P(\X + 11| > 1.5). 10.3. With > denoting the N0-l distribution function and Z NM distributed, show that (a)

292

The Normal Distribution, Central Limit Theorem

Chap. 10

10.4.

10.5.

10.6.

10.7.

10.8.

P(Z > z) = 1 - $(z); (b) P(Z > -z) = <f>(z); (c) P(-z < Z < z) = 2$(z) - 1; (d) P( ZI < Z < z2) = ^z^ + <t>(z2) 1, where z, z t , and z2 are nonnegative. Suppose that the length X of a random bolt produced by machine A is 1.2 cm, on the average, with a standard deviation of .3 cm. Assuming that the length X is normally distributed, find (a) P(X < 1.4); (b) P(X > 1.3); (c) P(\X - 1.2| > .2). Suppose that the number of brand X cornflakes in a box is ^2000.40000 distributed. Let Y be the number of flakes in four boxes. What is (a) (Y); (b) Var (Y)? (c) What is the distribution of Y? Find (d) P(Y < 8200); (e) P(|Y - 8000| > 300). X is the actual weight of a bag of concrete (labeled "80-lb bag"). Suppose that E(X) = 81 and o> = 2. Let Y be the average weight of 10 bags bought: Y = (X, + + X10)/10. (a) What is the distribution of Y assuming that each bag's weight is normally distributed? Find (b) P(Y ^ 80); (c) P(|Y - 81.5| < 1). Let X be N^ai distributed. Find (a) P(\X - u.| < a/2); (b) P(\X - \L \ < a); (c) P(\X - \IL\ < 1.5a). (d) Let the function g be defined by g() = P(\X - u, | < a). Show that g() = 23>(f) - 1. Plot g() as a function of t. (e) What is g'(0)? Actually show that an JV0jl random variable has mean O and variance 1. That is, assume that Z has density e~x2/2/V2~Tr. Show that

dx = O

(For the second integral use integration by parts: u = x, v' = xe *2'2.) 10.9. (Continuation) Show that the third and fourth moments of Z are (Z3) = O and (Z4) = 3. 10.10. Let X be N0j<j2 and Y be JV0>p2 distributed. Show that Z + Y is N0^ + distributed. 10.11. Let X be N^ and Y be NVi|)2 distributed. What are the distributions of X - u, and Y - vi Show that (X + Y) - (jx + v) is NO,^^ distributed using Problem 10.10. Conclude that X + Yis A^ +v>tr 2 +p 2 distributed. Extend this result to prove the boxed result before Example 5.

70.3 THE CENTRAL LIMIT THEOREM Consider a random sample of measurement {X,}f=l. Then the fact that the X's are identically distributed implies that they have a common distributionthe population distribution. In the preceding section we showed that if the population distribution is normal, then the sample mean X is also normally distributed. The central limit theorem states that even though the population distribution may be far from being normal, still for large sample size n, the distribution of the sample mean X is approximately normal with better approximations obtained the larger the sample size n. The central limit theorem is truly remarkable; it allows calculations of probabilities involving X even when the individual Xs have a very complicated and unknown distribution. Only the population mean u, and standard deviation o- are required to obtain the approximate distribution of X. Sec. 10.3 The Central Limit Theorem
293

The central limit theorem will be stated twice, once for sums and then sample means. Let X{, . . ., Xn be IIDRVs with common mean JJL and standard deviation a. Then the sum

Sn = X1
has mean and variance

Xn

E(Sa) = E(X1
Var (Sn) = ncr2 Thus (5,, - AZ|x)/Vcr is standardized.

Xn) =

The Central Limit Theorem for Sums -L, . . ., Xn are IIDRVs with common mean |JL and standard deviation CT. Let

Sn - Xl
Then the distribution of

z*A~ Va
approaches the N01 distribution as Thus Z = (Sn |X)/VCT will have a distribution that is approximately N0 for n large. How large is large enough? That depends on the individual distributions of the X/s as well as the accuracy required. The closer the distribution of X is to the bell-shaped normal curve, the smaller n needs to be to provide good accuracy. Specific vales of n will be given in the next two sections, but in general, surprisingly small (n > 20) vales of n provide reasonable accuracy for most applications. Example 7 Light bulbs are installed successively into a socket. Assume that each has a mean life of 2 months with a standard deviation of 1/4 month. Find P(40 bulbs last at least 7 years). Solution Let X denote the lifetime of the z'th bulb installed. Then n bulbs last a total time of Sn = X1 + + Xn. With n = 40, |x = 2, and cr = 1/4

SM - 4 0 - 2 V40- .25
is approximately W0jl distributed assuming that n = 40 is large enough.
The Normal Distribution, Central Limit Theorem Chap. 10

294

P(S40 2s 7 12 months) = P

S4n - 80

84 - 80

V4 .25

V4 .25

= P(Z > 2.530) = .0057 Example 8 (Continuatiori) How many bulbs n should be bought so that one can be 95% sure that the supply of n will last 5 years? Solution Note that n = 30 bulbs have a total expected life span of 30 2 = 60 months = 5 years. For general w, the total life span Sn has

E(Sn) = 2n

o-Sn = V .25

Using the Central Limit Theorem, we seek that valu of n so that

.95 = P(Sn & 60)

Sn - 2n > 60 - 2n \V .25 ^ V .25

Thus a table look-up implies that

60 - 2n 7= = -1.645 V - .25
This is a quadratic equation in root n: 60 - 2n = - 1.645V .25 = -.4113 V

2n - .4113 V - 60 = O
n =

.4113 V(.4113)2 + 4 2^60 = -5.375, 5.581 2-2

Eliminating the negative root, we have

n = 31.15
So we should buy 32 bulbs. In this example the number 32 is fairly large, so that the central limit theorem can apply. Purely on the basis of the number 32 calculated by means o/the central limit theorem are we afterward justified in using the central limit theorem. The logic of this is strange, but nonetheless valid. A proof of the central limit theorem is beyond the scope of this book, but

Sec. 10.3

The Central Limit Theorem

295

the next two examples show how the bell-shaped density curve emerges in specific cases: Example 9 Suppose that each X is uniformly distributed on [O, 1]. Then |o. = 1/2 and cr2 = 1/12 from the results of Chapter 9. Thus

Z=

Sn - n/2

will be approximately N01 distributed for large. For = 50, 500 simulations of S50 were generated by computer and a histogram of total rea 1 was drawn from the 500 vales of (550 - 25)/V50/12. The N0 l density function 4> is overlaid in Fig. 10.8.

Example 10 Let T!, . . . , Tn be IIDRVs, each exponentially distributed with parameter X. Then Sn = 7\ + + Tn is gamma distributed. Thus the central limit theorem claims that Sn, once standardized, has a density that is approximately Af0>1 for n large. Since E(T) = I/A and Var (T) = 1/X2 from Chapter 9,

Z=

Sn -

296

The Normal Distn'bution, Central mit Theorem

Chap. 10

s in specific

is standardizad. For several vales of n Fig. 10.9 shows the density of Z with the N0<1 density superimposed:

2 and a2 =

simulations drawn from overlaid in

Figure 10.9

The central limit theorem can be phrased in terms oftie sample mean rather than the sum Sn. Note that with Sn = Xl + + Xn, X = SJn. Therefore,
Sn -

X -

is standardized. The Central Limit Theorem for Sample Means rameter X. t theorem ' N01 for n Xl, . . . , Xn are IIDRVs with common mean |x, standard deviation a, and sample mean X. Then the distribution of
Z =
X-

er/V

approaches the N01 distribution as n o.

Chap. 10

Sec. 10.3

The Central Limit Theorem

297

Example 11 (Continuation of Example 7) Let S be the average life of a package of 25 light bulbs. (a) Find P(S > 1.9 months). (b) Also find an interval (2 - c, 2 + c) about the mean JJL = 2 so that we can be 95% sure that the average lifetime S will fall into this interval. Solution (a) Applying the central limit theorem,
Z =

5-2 .25/5
1.9 - 2 .25/5

is approximately N0;1 distributed. Thus

P(S > 1.9) = P(Z>

= P(Z> -2)

= .9772 (b) To find c, note that

.95 = P(2 - c < S < 2 + c)
70.4

.25/5
rea = 0.95

.25/57

rea to left of here s 0.975

Figure 10.10

Therefore, c must satisfy

.25/5

= 1.96

c = .098
Thus 95% of the packages with 25 bulbs will have average life in (1.902, 2.098).

PROBLEMS 10.12. Suppose that wildflowers occur at distances along a path X apart where E(X) = 100 feet and <jx = 50 feet. Let Sn = distance until the nih clump of wildflowers. 5n

298

The Normal Distribution, Central Limit Theorem

Chap. 10

ind P(S > 1.9

i |x = 2 so that iterval.

10.13.

10.14.

10.15.

10.16.

= Xl + + Xn. Find (a) P(30 clumps within 2700 feet of start) = F(530 =s 2700); (b) P(fewer than 15 clumps within 1800 feet of start) = P(S5 > 1800). (Continuatiori) Find the largest number n so that one can be 90% sure that there will be at least n clumps within 1 mile of the start of the path. That is, find the largest n so that .90 =s P(Sn =s 5280). (Continuation) Suppose that a = 10 feet rather than 50 feet. Now find the n in Problem 10.13. Is the result reasonable? Should a decrease in a cause an increase or a decrease in n? Benjy can hit a nail an average of 1/4 centimeter per hammer blow with a standard deviation of .1 centimeter. To be 90% sure of driving a 5-centimeter nail all the way, how many hammer blows must Benjy do? Find n so that .90 s P(Sn ^ 5), where Sn is the distance the nail is driven in n blows. (First show that .90 = P(Sn 3= 5) implies (5 - n/4)/.lV = -1.28.) Batteries last an average of 2 months with a standard deviation of 10 days (assume 30 days/month). Twenty-five batteries are bought. Find (a) P(average life of all 25 > 61 days); (b) P(average life differs from 2 months by more than 3 days). (c) How many batteries n must be bought so that the average life of all n is 2 months to within 1 day with probability at least .95?

10.4 THE CENTRAL LIMIT THEOREM APPLIED TO THE BINOMIAL DISTRIBUTION Perform n Bernoulli triis each with probability p of success. With X = 1, O if the z'th trial results in success, failure, respective ly, the sum

sn = x,
is binomially distributed. From Chapter 9,

xn

JJL = E(X,) = p o-2 = Var (X,) = pq

Consequently, the central limit theorem applied to Sn implies: If Sn is binomially distributed n, p, then for n large, 1.902, 2.098).

is approximately N01 distributed. Example 12

re E(X) = 100 ildflowers. S

In n = 100 flips of a fair coin, find P(45 = S s 55), where S = 5100 is the total number of heads.

Chap. 10

Sec. 10.4

The Central Limit Theorem Applied to the Binomial Distribution

299

Solution 1 np = 100/2 = 50 and Vpq = VlOO/4 = 5.

= P(-l =s Z +1)
= .6826 Solution 2 Since it is impossible to obtain a nonintegral number of heads, (answer) = P(44 < S < 56)
6

5-50

6^

= P(-1.2 < Z< 1.2) = .7698 Solution 3 Or perhaps since the discrete random variable Sn is being approximated by a continuous one, the boundary of the event might better be taken at half-integer points: (answer) = P(44j =s S = 55)
5
5 ^ 5

= P(-l.l Z 1.1) = .7286 Intuition suggests that Solution 3 is best. The exact probability is
P(45
55) = = .7281

Continuity Correction

Let j < k be integers and let S be an integer-valued random variable. In using the central limit theorem to find P(j ^ S * k), apply the theorem to
Pj Example 13

^k + ~

Of the bolts made by machine A, 10% are defective. The first 1000 produced after a tune-up produced only 80 defective bolts. (Expected number 100.) Is this sufficient evidence to conclude that the tune-up improved performance? Solution Each of the 1000 tested bolts constitutes one Bernoulli trial. Under
300

The Normal Distribution, Central Limit Theorem

Chap. 10

the assumption that the tune-up had no effect, the number of defective bolts S is binomially distributed, n = 1000, p = .1. Therefore,
S - 100 V9

is approximately JV0jl distributed. Assuming that the tune-up had no effect, P(S 80) = P(S 80.5)
8Q 5

- "

-2.06)
= .0197

using the continuity correction to obtain the first equality. Conclusin: Assuming that the tune-up had no effect, so few defective bolts would have been produced in about 2% of all samples of size 100. We are justified, tentatively, in assuming that the tune-up worked. The central limit theorem can be applied to the sample mean X rather than the total number of successes Sn in n Bernoulli triis. In fact, X = SJn is the fraction of the number of triis resulting in success. n Bernoulli triis are performed. X equals the fraction of them which resulted in success. Then
Z =

X-p

is approximately W 0jl distributed for n large.

Example 14

A pollster wants to find the proportion p of adults who favor gun control. How many adults n must be sampled to find the valu o p? In more detail: The exact valu of p cannot be known unless all the adults in the entire population are sampled. How many n must be sampled to find/? to within one-one hundredth? Again, depending on the exact composition of the particular sample of size n, the estmate of p will differ; some samples will even have an estmate of p equal to 0. (No one favored gun control in that sample.) But (and the final_restatement of the question) how many n must be sampled so that the fraction X in the sample who favored gun control is within .01 of p with probability .95? Solution With p = true fraction of the adults who favor gun control, Sn = number of adults favoring gun control in a sample of size n is binomially distributed.
Sec. 10.4 The Central Limit Theorem Applied to the Binomial Distribution 301

(1/2,1/4)

1/2

Figure 10.11

Thus the sample mean X = SJn is in fact an estmate of p. n satisfies

.95 = P(\X - p\ =s .01) .01

X - p /pqln

.01

Ipqln
1.96^

\/pqln/

where Z is AT0jl distributed. A table look-up implies that

At this point it is not altogether clear how to proceed. We wanted a specific valu of n. If the pollster had hired us to compute the valu of n and tell him how many adults need to be sampled, we cannot very well present (196^/pq)2 as the answer. If we do, the pollster will simply say that in order to find the specific number of adults to sample, pq = p(l p) must be known. But this entails knowing p, which is what the pollster really wants to know in the first place. If our answer is (196^/pq)2, our answer is useless! Consider the graph of the function/(p) = pq = p(l p) in Fig. 10.11. pq = p(l p) is a quadratic in p with mximum valu 1/4. [Differentiate p(l - p) and set derivative equal to 0.] Therefore, in order that the number n be independent of the true proportion p of adults who favor gun control, the number to be sampled is n = 1962 pq * 1962 ^ = 9604 Conclusin: To be 95% sure of "capturing" the correct proportion p, sample about 9600 adults. Fewer may have been necessary, but this many will always (regardless of the true valu of p) be enough. For p cise to 1/2, the distribution of binomially distributed Sa will be cise to symmetric with respect to its mean np. Thus given a valu of n, the normal approximation will be better the closer p is to 1/2. In general, The normal approximation to the binomial distribution is accurate for npq > 5.
302
The Normal Distribution, Central Limit Theorem Chap. 10

The valu 5 is somewhat arbitrary. But consider this comparison of vales: For n = 21 and p = 1/2 = q, let S be the number of successes in n Bernoulli triis. Then S is binomially distributed with np = 10.5, npq = 5.25. The exact probability that S = j is
'21

i
Using the central limit theorem, the approximation is P(S = j) = P(; - . 5 < S < / + .5)

where Z is N0-1 distributed.

P(S = j )

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

.0006 . 0 0 2 9 . 0 0 9 7 ' .0259 .0554 . 0 9 7 0 .1402 .1682 .1682 .1402 .0970 .0554 .0259 .0097 .0029 .0006

.0009 .0033 .0101 .0259 .0548 .0948 .1399 .1687 .1687 .1399 .0948 .0548 .0259 .0101 .0033 . 0 0 0 9

PROBLEMS 10.17. Let S be binomially distributed with n = 24, p = .6. Compute the exact valu of P(S = ;') and compare it with the approximation using the central limit theorem to compute P(j - .5 < 5 < ; + .5) for / = (a) 10; (b) 15; (c) 20. Note that

where Z is NM since np = 24 .6 = 14.4 and npq = 24 .6 .4 = 5.76. To compute the approximation, compute the right-hand side for/ = 10, 15, and 20. 10.18. Suppose that 40 people in 100 favor candidate C. In a randomly selected group of Chap. 10 Problems
303

100 people, let S = #(who favor C). Find the central limit theorem approximations to (a) P(38 S 42); (b) P(37 < S < 43); (c) P(3712 =s S *s 42 i). 10.19. Roll a fair die 1000 times. Let X equal the fraction of the times that either a 1 or a 2 occurs. Find (a) P(X < 340); (b) P(330 < X 337). Make sure to use the continuity correction. 10.20. Let S be the number of success in 10,000 Bernoulli triis. What is E(S)7 Find P(10,000p - 20 < S < 10,000p + 20) for/? = (a) .1; (b) .25; (c) .50; (d) .75; and (e) .9. (f) Plot this probability as a function of p. 10.21. Perform n Bernoulli triis with p = 1/2. Use the central lmit theorem to find an interval about n!2(n/2 - c, n/2 + c) so that the actual number of successes will fall into this interval with probability .90. What is the length of this interval? How does it depend on ni 10.22. Let Sn = #(heads on n flips on a fair coin). How many flips must be performed so that with probability at least .95 the average number of heads per flip is 1/2 to within .01? That is, solve for n in

.95 = P

< .01

10.23. A marketing research analyst wants to find the proportion p of people who like brand X cereal. He/she sets up shop in a busy mal, n people are interviewed and Sn are found to like brand X cereal. Given p, why is Sn binomially distributed? Find n so that the analyst can be 99% sure that SJn is within .1 of p. 10.24. A certain fraction p of seeds cast to the wind germnate. A sample of n seeds were scattered and Sn germinated. If X = SJn is taken to be the estimator for/?, how large should n be so that X will be within .05 of p with probability .95? Obtain n as a function of p and graph it.

10.5 THE CENTRAL LIMIT THEOREM APPLIED TO THE POISSON DISTRIBUTION The normal distribution also approximates the Poisson distribution for a large valu of the parameter X. To see how this is so, let X be Poisson distributed with parameter A > 0. Since the mean and the variance of X are both equal to the parameter A,

Z=

X -A

VA

is standardized. Consider this model: Calis arrive at an exchange according to a Poisson stream with X = 1 cali per minute, on the average. Let X = #(calls registered in the interval (/ - I,/]) Then the number of calis registered in (O, t] is

X = Xl + + X,
304

The Normal Distribution, Central Limit Theorem

Chap. 10

But

u. = E(X.) = X = 1 a2 = Var (X) = X = 1

since each X is itself Poisson distributed with parameter X = 1. Also, the X?s are independent by the first Poisson postlate, which states that the number of calis registered in nonoverlapping time intervals are independent. Thus we can conclude: 1. X is Poisson distributed with parameter fX = t. 2. (X - t)/\/i = (X - X)/VfX is approximately N0 distributed for large t by the central limit theorem. Now set = A and the general result can be phrased without reference to the reasoningthe modelused to derive it: If X is Poisson distributed with parameter A, then
X

~A VA

is approximately W0,i distributed for A large. Example 15 Calis arrive at the rate of X = 2 calis per minute. In an 8-hour day, therefore, 2 6 0 - 8 = 960 calis are expected. Let X = #(calls actually registered). Find
P(X < 1000) P(X > 900)

and an interval around 960[960 - c, 960 + c]so that with probability 1/2, X will fall in this interval. Solution A = E(X) = 960. Therefore, Z = (X - 960/V960 is approximately JV0jl distributed. Using the continuity correction, P(X < 1000) = P(X ^ 999.5)
= P

<X - 960 999.5 - 960^ /960 /960

= P(Z = 1.275) = .8997 P(Z > 900) = P(Z s= 900.5)

= P

^ - 960 900.5 - 960 /96 %0 = -1.920)

= .9726
Sec. 10.5 The Central Limit Theorem Applied to the Poisson Distribution 305

Finally, the number c satisfies .5 = P(960 - c = X ^ 960 + c)

= P Therefore,

c /96 = .675

c /960

V960

c = 20.91
Thus with 50% probability there will be 940 or 941 or . . . or 980 calis. Example 16 A snack bar sells hot dogs at the rate of 1 per every 7 minutes. How many minutes must the snack bar stay open so that with probability 90% all of the 6 dozen hot dogs are sold? Solution Calis coming into an exchange hot dogs being sold. The same three Poisson postlales apply to each. Henee the number of hot dogs sold in minutes X, is Poisson distributed with parameter Kt = til . (In / minutes we expect to sell til hot dogs.) We seek that valu of t that solves the equation

.90 = P(Xt > 6 12) = P(X, 3= 72) With A = til the central limit theorem states that X, - til

Vt
is approximately N01 distributed for large t.

.90 = P(Xt 3= 72) X, - til til

= P

12 - til tll

12 -

Vt
72 - til

A table look-up allows us to continu:

Vt
- - 1.28

-1.28

- - 72 = O

- - .4838 Vt - 12 = O .4838 V(A838)2 + 4 72/7 = -20.82, 24.21 2/7

305
The Normal Distribution, Central Limit Theorem Chap. 10

Discarding the negative root gives us t = 24.212 = 586.1 minutes = 91 hours Note that in 586.1 minutes the snack bar can expect to sell

586.1
7

= 83.7 = 1 dozen

hot dogs. But according to our calculations, we can be only 90% sure of sellinj; 6 dozen. calis. The normal approximation to the Poisson distribution is accurate for A > 15. The number 15 is somewhat arbitrary. Suppose that X is Poisson distributed with A = 16. The following table compares the exact probabilities P(X = /) with the approximation using the central limit theorem with the continuity correction. Note that
-

many minutes e 6 dozen hot d. The same dogs sold in / utes we expect on

is the exact probability; the approximation is

+

P(X = /) 1) = i\l where Z is N01.

-J ^ ^ -- J ' "V

^ z, ^

-5~

J) ~ -5 ~ 16 _ ^ ^ + -5 ~ 16\

/ 5
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Chap. 10 Sec. 10.5

P(X = j)

4
.0023 .0045 .0080 .0136 .0217 .0325 .0458 .0605 .0752 .0889 .0954 .0995 .0954 .0889 .0752 .0605 .0458 .0325 .0217 .0136

.0010 .0026 .0060 .0120 .0213 .0341 . 0 4 9 6 .0661 .0814 .0930 .0992 .0992 .0934 .0830 .0699 .0559 .0426 .0310 .0216 .0144

The Central Limit Theorem Applied to the Poisson Distribution

307

PROBLEMS 10.25. On a given day 1000 cars can be expected to pass a certain toll booth. Why should the actual number of cars X be Poisson distributed with parameter A = 1000? Use the normal approximation to find (a) P(X > 960); (b) P(X = 1050); (c) F(980 < X = 1020). Make sure to use the continuity correction. 10.26. Let the parameter A be 15 for Poisson distributed X. Compute P(X = /) and the normal approximation to P(j - 1/2 < X < j + 1/2) for/ = 10, 15, and 20. 10.27. Raindrops collect on the window pane at the average rate of X = 20 per minute, (a) What is the exact distribution of N, = #(drops in t minutes)? Find (b) P(N5 *s 105); (c) P(91 <N5< 109); (d) P(92 =s N5 =s 108); (e) P(91 = N5 ^ 108) using the central limit theorem. 10.28. A business has 5 telephone lines. The first rings every minute on the average, the second every 1/2 minute, the thrd every 2 minutes, the fourth every 5 minutes, and the fifth every 1/10 minute, on the average. Let X, = #(calls in [O, ]). (a) What is the exact distribution of X,l Find (b) E(Xea) and (c) P(X60 < 850) using the normal approximation. (d) Find a number c so that with 95% probability the number of calis will be less than c in 1 hour. 10.29. Let Xt = #(events in [O, t\) for a Poisson stream with parameter X. Suppose that X = 100. (a) What is the exact distribution of X,l Find g(t) = P(X, =s 2000) for these vales of t: (b) 19; (c) 19.5; (d) 20; (e) 20.5; (f) 21; (g) Plot g(i) as a function of . 10.30. A switchboard receives orders at the rate of 1 every 2 minutes, (a) What is the exact distribution of X, = #(orders in [O, ])? (b) How long must the switchboard plan on staying open so that with 90% probability at least 100 orders will have been made in [O, ]?

10.6 CHEBYSHEV'S INEQUALITY AND THE LAW OF LARGE NUMBERS

Of less computational applicability than the central limit theorem, but with as much theoretical interest, is the law of large numbers. In Chapter 1 the frequency interpretation of probability was used as the intuitive ground for developing the entire axiomatic structure of probability theory. In Chapter 4 the frequency interpretation was again used to develop the definition of the expectation. The law of large numbers is actually the frequency interpretation phrased as a theorem. In stating and proving the law of large numbers, we come full circle: The whole structure of probability is based on the frequency interpretation the axioms of which can be used to prove the frequency interpretation. The law of large numbers uses a technical tool called Chebyshev's inequality. Pafnutii Lvovich Chebyshev (1821-1894) performed path-breaking work in many reas of applied mathematics. Under his supervisin Russia carne into the forefront of probability theory. He felt that only mathematics that arse out of real

308

The Normal Distribution, Central Limit Theorem

Chap. 10

applications was of valu, although he was particularly able in generalizing from particular cases to general theories. Let W be a random variable with mean 0. Although the expectation of W is O, any single realization of W may be very far from 0. The variance measures how far the vales of W are dispersed from 0. The larger the valu of Var (W), the less surprised one would be to find the actual valu of W far from 0. Chebyshev's inequality makes this idea precise. The question is: Given the variance a2, how cise to u. = O are the vales of W likely to be? To try to answer this, fix a number e > O and let us ask for the probability that W is farther than e away from its mean u, = 0. In fact, we expect that this probability P(\W\ 3= e) should get larger as Var (W) = a2 gets larger; for then the vales of W will be spread over a larger interval. Let us suppose that W has density/. (A similar derivation applies when W is discrete rather than continuous.) Then for fixed number e > O,
P(\W\ s* e) = i

f(x) dx

~J(x)dx

-2f(x)dx

E(W2)
cr

Note that the first inequality follows from the fact that the interval of integration is the points x where x2 ^ e2, and henee the integrand can only be larger if e2 is replaced by x2 in the numerator. The second inequality follows from increasing the interval of integration from the points x where x2 3= e2 to the entire line from -oo to 0. The second-to-last equality follows from the definition of E(W2). Henee we nave shown that if E(W) = O and given any positive number e, the event that W differs by at least E from 'O is bounded:
P(\W\

& E) * ^

Sec. 10.6

Chebyshev's Inequality and the Law of Large Numbers

309

Now suppose that X is any random variable; and let JJL = E(X). X - (A has E(W) = O, so the abo ve applies to W. To summarize:

Then W

Chebyshev's Inequality Let X have finite mean |JL and finite variance tr2. Then for e > O a fixed number, the chance that X differs by at least E from its mean is bounded:
P(\X - (i| 3* 8) * ^

In terms of the complementary event,

P(\X U, < 6) ^ 1 ~

Example 17 Boxes of bolts have an average of 100 bolts with a standard deviation of 3. Find the probability that the number of bolts in the box I buy is between 95 and 105. Solution 1: Using Chebyshev's inequality. Let X be the number of bolts in the box. Then

u, = 100
Henee

a = 3

P(95 < X < 105) = P(\X - 100| < 5) 2= 1 -

With at least chance 14/25 = .56, the box contains between 95 and 105 bolts. Solution 2: Using the central theorem Assuming that the central limit theorem applies to the random variable X (that XK due to many small causes, in other words), then X is approximately NIQ09 distributed. Henee P(95 < X < 105) = P(95.5 = X *z 104.5)

= P

'95.5 - 100

104.5 - 100N

1.5)
= .8664 For large sample size the probability is approximated to better accuracy using the central limit theorem than using Chebyshev's inequality. This is so since Chebyshev's inequality is a general result that holds true for every distribution with the same mean and variance. //, however, the distribution is known to be approximately normal, then the specific details of that distribution can be used. But the weakness of Chebyshev's inequality due to its generality is precisely
310

The Normal Distribution, Central Limit Theorem

Chap. 10

what is required to prove the law of large numbers and the consistency of the frequency interpretation of probability. Let {X,}f=1 be a random sample. Then the Xs are IIDRVs with common mean (JL and variance a2. One intuitively expects that the sample mean X should be cise to the population mean x for n large. Mathematically, this is expressed by
lim X = lim = u.

where
Sn = X, +

is the sum. More precisely, we will show that the probability that X differs by any nonzero amount from |JL tends to O as n becomes large. What is surprising is not that this result holds (it must hold in any reasonable theory of probability), but that our mathematical machinery allows us to prove it. Let e > O be a fixed number. By independence of the sequence Xlt . . . , Xn in the random sample, E(Sn) = n\L
Var (5B) = a2

Chebyshev's inequality applied to Sn states that

P(\Sn - |x| ^ ne)

Var(5J
(e)
2

a2

a2

In terms of the sample mean X,

P(\X - ji| s* E) = P
= P(\Sn CT2
ntL\

^ m)

Now let n o. Then the bound on the right tends to O and the following conclusin follows: Law of Large Numbers Let {AT,}"=! be a random sample with common mean |x and variance o-2. Let
o _ y _i_ . . . _i_ y n A\ ^ ^ -A-n

Then
Sn_

n
as n oo for each fixed e > 0.
10.6

Chebyshev's Inequality and the Law of Large Numbers

311

Example 18 Perform n Bernoulli triis with probability p of success. The fraction of the times resulting in success is

X = ~(X1+--- + Xn) = ^
where X is 1, O with probability/, q respectively. Since JUL = E(X) = p,

P(\X - />( s e) - O
as n 0 for each fixed e > 0.

To show how the law of large numbers leads to the frequency interpretation of probability, consider an event A associated with a random experiment. Perform the random experiment a large number n of times. Define random variable X by l O i'th experiment results in event A otherwise

Then Sn = Xl + + Xn is the total number of times A has occurred on all n experiments. Since
E(Xi) = 1 P(X = 1) + O P(X = 0) = P(X)

the law of large numbers states that

<

as n 0 for fixed e > 0. The law of large numbers says this: A random experiment is performed and X1 is measured, then the experiment is repeated and X2 is measured, and so on.
SJn

M -e

Figure 10.12

312

The Normal Distn'bution, Central Limit Theorem

Chap. 10

he times

With Sn = X1 + + Xn, Snln is the sample average of the X's over the frst n experiments. The law of large numbers says that for n large, we can be as sure as we like that SJn is cise to |x. That is, if an error e is set, the chance that Sn/n differs from u, by more than the error e goes to O as n . Or, stated differently, the chance that SJn is withn e of ^ tenas to 1 as n > o. (In Fig. 10.12, with large probability the valu of SJn will be within the horizontal lines distance e from JJL.) If each repetition of the random experiment takes 1 minute, then SJn is the average of the X vales over the course of the first n minutes. Thus the time average SJn approaches the population average [i as the time tends to .
Example 19

etation 'erform

Consider a Poisson stream with parameter X. With TV, = #(calls in [O, t\), E(Nt) = X and Var (Nt) = X since the counting variable Nt is Poisson distributed with parameter Xt. By Chebyshev's inequality, therefore,
TV
' - X

.= P(\N, - X/| > e/)

^ Var (Nt) (ef)2

n all n

X
(e/)2

d and ;o on.

-* O as t > oo. Thus the temporal average number of calis per unit time N,/t tends to X as the time .

10.7 THE CHI-SQUARE DISTRIBUTION AND SAMPLING FROM THE NORMAL DISTRIBUTION (OPTIONAL) AssumethroughoutthatZj, . . . , Zn are IIDRVs, each with the normal distribution with mean O and variance 1. Then by definition, the random variable

Yn = Z\ + + Z*
has the chi-square distribution with n degrees of freedom (often denoted "x2 distribution," using the Greek lowercase letter chi). Note that Yn is a continuous random variable concentrated on the positive real axis (O, ). Why?
Example 20

Find the density /i for the x2 random variable Yl with one degree of freedom. Solution /i(y) = O for y < 0. Yl = Z2, where Z is 7V0>1. Thus the distri-

10

Sec. 10.7

The Chi-Square Distribution and Sampling

313

bution function for Y1 is

= P(Z2 ^ y)

= P(\z\ ^ Vy) i

[^ -y TT Jo
Differentiate to find the density function:

using the chain rule to differentiate the integral in which the top limit is a function o y. To summarize:

Example 21

Find the density for Y2. Solution Use the convolution formula for the density of a sum of independen! random variables derived in Section 8.5. Y2 = Z\ + Z\, where Zf and Z\ are each x2 distributed with one degree offreedom', their density functions were derived in Example 20. Consequently, since they are concentrated on (O, <*>),

/20) =

- O dt

dt
2lT

JO

/2 r/2 (y -

Now use the substitution t = u2 in the integral:

du

2TT

o Vy - u2
Chap. 10

314

The Normal Distribution, Central Limit Theorem

!_
TT

VJ B _ 0
1 - sin"1 0)

1 TT

where the integration of 1/Vy - u2 was done using a table of integris. density function you may recognize: Mysteriously enough: Y2 has the exponential density with parameter 1/2:
O /2

This

y <O
y > 0

For degrees of freedom > 2, the exact density function for Yn is derived in the problems. To summarize: The density function for the chi-square distributed random variable Yn with n degrees of freedom is
O

y <O

,(30 =
where Cn is a constant. valu of Cn.) (See Problems 10.31 to 10.33 for the exact

Fig. 10.13 depicts graphs of the chi-square density functions for various vales of the parameter n. Here is a methodfor simulating an 7Voa distributed random variable using the X2 distribution with two degrees of freedom: Let a two-dimensional coordnate system have horizontal and vertical axes labeled s and t, respectively. A point is chosen at random with coordinates (S, T). Assume that the random variables S and T are independen!, each N0 distributed. Then the distance of the point from the origin is R = VS2 + T2. From Example 21 we know that the random variable R2 is x2 distributed with two degrees of freedom, which is to say that R2 is exponentially distributed with parameter 1/2. Let the angle > be the angle O =s <I> < 2-77 the line segment from the origin to the point (S, T) subtends with rsped to the positive horizontal axis as in Fig. 10.14. Let us show that 4> is uniformly distributed on the interval [O, 2ir). In fact, if u is a fixed number with O s u < 2tr, then <J> =s u f and only if (S, T) is contained in the wedge A of Fig. 10.15.
Sec. 10.7 The Chi-Square Distribution and Sampling
315

a I

316

The Normal Distribution, Central Limit Theorem

Chap. 10

Figure 10.14

Figure 10.15

Thus the distribution function for the random angle <I> is

u) = P((S, T) e A)

dsdt

Now switch to polar coordinates: "

s = r eos 0
r =s + t
2 2 2

t = r sin 0
ds dt = rdrdQ
e-ri/2

2ir Jo Jo
f) ^H

O
FH

Jo
2-TT

1 d%

which proves that <I> is uniformly distributed on the interval [O, 2n) since its distribution function has that form. Let us summarize: If a point (S, T) is chosen so that S and T are independent, each N01 distributed, then R2 is exponentially distributed with parameter 1/2 and $ is uniformly distributed on [O, 2ir), where R is the distance from the origin and 4> is the angle subtended by (S, T). To simlate an NOA valu, tura this reasoning around: First simlate an exponentially distributed random variable E with parameter 1/2 and a uniformly distributed random variable U on [O, 2ir) and then set

S = VE eos U

Sec. 10.7

The Chi-Square Distribution and Sampling

317

From Chapter 7, Example 15, E is set equal to -2 In (1 - RND), where RND is uniform on [O, 1). To simlate an N0>1 valu X, first simlate RNDj and RND2 uniformly distributed in [O, 1) and set X = V-21n(l - RNDj) eos (2ir RND2)

Example 22 Write a program to simlate 100 jV0>1 vales. Solution The body of the program is
begin SEED:= 0;
for l:= 1 to 100 do begin SEED:= (MULT*SEED + ADOR) mod NORM; RND: = SEED/NORM; E:= - 2*ln(1-RND);

SEED:= (MULT*SEED + ADOR) mod NORM; RND:= SEED/NORM; U:= 2*3.14159*RND; X:= (E**0.5)*cos(U); write(X:10:4); end; end.

PROBLEMS 10.31. (Derivation of \2 Density with n Degrees of Freedom for n = 2k Even) Show that

Y2k = (z? + z) + + (zi*., + zy

is the sum of k independent random variables, each exponentially distributed with parameter 1/2 by Example 21. Conclude that Y2k is gamma distributed with n = k and X = 1/2. Thus conclude that

/oo =

y <O

\2k(k - 1)!

>- > O

318

The Normal Distribution, Central Limit Theorem

Chap. 10

e RND

10.32. (Derivation of \2 Density with n Degrees of Freedom for n = 2k + 1 Oda) Show that

Y 2k+l Y 72 -* 1k + ^
1 T

dis-

where Y2k and Z are independent, Z N0 1 distributed, and Y2k with the density of Problem 10.31. Conclude that the density for Yn is
fn(y) = Jo \

- u)-ll2du where C is a constant. Note that the integral in the last expression is itselfa constant independen! of>>. Thus
.,-y/2

for some constant Cn. Note that n = 2k + 1, so k = (n - 1)12. 10.33. To find Cn of Problem 10.32, show that
J2)-lp-y/2 dy C-' = />

= (n- 2)C-_>2 What is Ct? Successively find C3, C5, and then Cn for n odd. 10.34. (The Rayleigh Distribution) (a) Let R2 be x2 distributed with two degrees of freedom. Find the density function for R. (b) Let S = aR. Find the density function for S. 10.35. Let Ya be x2 distributed with n degrees of freedom. Show that E(Yn) = n and Var (Yn) = 2n. 10.36. Write a program that will input N, then genrate 100 simulated vales of a x2 random variable with N degrees of freedom. Use the technique of Example 22 to genrate N01 vales first.

rith = k

Chap.

10

Problems

319