Jump to content

Probability/Joint Distributions and Independence

From Wikibooks, open books for an open world


Motivation

[edit | edit source]

Suppose we are given a pmf of a discrete random variable and a pmf of a discrete random variable . For example, We cannot tell the relationship between and with only such information. They may be related or not related.

For example, the random variable may be defined as if head comes up and otherwise from tossing a fair coin, and the random variable may be defined as if head comes up and otherwise from tossing the coin another time. In this case, and are unrelated.

Another possibility is that the random variable is defined as if head comes up in the first coin tossing, and otherwise. In this case, and are related.

Yet, in the above two examples, the pmf of and are exactly the same.

Therefore, to tell the relationship between and , we define the joint cumulative distribution function, or joint cdf.

Joint distributions

[edit | edit source]

Definition. (Joint cumulative distribution function) Let be random variables defined on a sample space . The joint cumulative distribution function (cdf) of random variables is

Sometimes, we may want to know the random behaviour in one of the random variables involved in a joint cdf. We can do this by computing the marginal cdf from joint cdf. The definition of marginal cdf is as follows:

Definition. (Marginal cumulative distribution function) The cumulative distribution function of each random variable is marginal cumulative distribution function (cdf) of which is a member in the random variables .

Remark. Actually, the marginal cdf of is simply the cdf of (which is in one variable). We have already discussed this kind of cdf in previous chapters.

Proposition. (Obtaining marginal cdf from joint cdf) Given a joint cdf , the marginal cdf of is

Proof. When we set the arguments other than -th argument to be , e.g. , the joint cdf becomes

Remark. In general, we cannot deduce the joint cdf from a given set of marginal cdf's.

Example. Consider the joint cdf of random variables and : The marginal cdf of is

Similar to the one-variable case, we have joint pmf and joint pdf. Also, analogously, we have marginal pmf and marginal pdf.

Definition. (Joint probability mass function) The joint probability mass function (joint pmf) of is

Definition. (Marginal probability mass function) The marginal probability mass function (marginal pmf) of each which is a member of the random variables is

Proposition. (Obtaining marginal pmf from joint pmf) For discrete random variables with joint pmf , the marginal pmf of is

Proof. Consider the case in which there are only two random variables, say and . Then, we have Similarly, in general case, we have Then, we perform similar process on each of the other variables ( left), with one extra summation sign added for each process. Thus, in total we will have summation sign, and we will finally get the desired result.

Remark. This process may sometimes be called 'summing over each possible value of other variables'.

Example. Suppose we throw a fair six-faced dice two times. Let be the number facing up in the first throw, and be the number facing up in the second throw. Then, the joint pmf of is in which , and otherwise. Also, the marginal pmf of is in which , and otherwise.

By symmetry (replace all with and replace all with ), the marginal pmf of is in which , and otherwise.

Clipboard

Exercise. Suppose there are two red balls and one blue ball in a box, and we draw two balls one by one from the box with replacement. Let if the ball from the first draw is red, and otherwise. Let if the ball from the second draw is red, and otherwise.

1 Calculate the marginal pmf of .

2 Calculate the joint pmf of .

)
)
)


Clipboard

Exercise. Recall the example in the motivation section.

(a) Suppose we toss a fair coin twice. Let and . Show that joint pmf of is

(b) Suppose we toss a fair coin once. Let and . Show that joint pmf of is

(c) Show that marginal pmf of and are in each of the situations in (a) and (b). (Hint: for part (b), we need to put value in the variable in the indicator)


Proof.

(a) Since the support of is , the joint pmf of is

(b) Since the support of is , the joint pmf of is

(c) Part (a): marginal pmf of is and marginal pmf of is

Part (b): The marginal pmf of is Similarly, the marginal pmf of is

For jointly continuous random variables, the definition is generalized version of the one for continuous random variables (univariate case).

Definition. (Jointly continuous random variable) Random variables are jointly continuous if for some nonnegative function .

Remark.

  • The function is the joint probability density function (joint pdf) of .
  • Similarly, can be interpreted as the probability over the 'infinitesimal' region , and can be interpreted as the density of the probability over that 'infinitesimal' region, i.e. , intuitively and non-rigourously.
  • By setting , the cdf

which is similar to the univariate case.

Definition. (Marginal probability density function) The pdf of each which is a member of the random variables is the marginal probability density function (marginal pdf) of .

Proposition. (Obtaining marginal pdf from joint pdf) For continuous random variables with joint pdf , the marginal pdf of is

Proof. Recall the proposition about obtaining marginal cdf from joint cdf. We have

Proposition. (Obtaining joint pdf from joint cdf) If a joint cdf of jointly continuous random variables has each partial deriviative at , then the joint pdf is

Proof. It follows from using fundamental theorem of calculus times.

Example. If the joint pdf of jointly continuous random variable is the marginal pdf of is Also,

Clipboard

Exercise. Let and be jointly continuous random variables. Consider the joint cdf of :

1 Calculate the joint pdf of .

2 Calculate the marginal pdf of .



Independence

[edit | edit source]

Recall that multiple events are independent if the probability for the intersection of them equals the product of probabilities of each event, by definition. Since is also an event, we have a natural definition of independence for random variables as follows:

Definition. (Independence of random variables) Random variables are independent if for each and for each subset .

Remark. Under this condition, the events are independent.

Theorem. (Alternative condition for independence of random variables) Random variables are independent if and only if the joint cdf of or the joint pdf or pmf of for each .

Proof. Partial:

Only if part: If random variables are independent, for each and for each subset . Setting , and we have Thus, we obtain the result for the joint cdf part.

For the joint pdf part,

Remark.

  • That is, if joint cdf (joint pdf (pmf)) can be factorized as the product of marginal cdf's (marginal pdf's (pmf's))
  • Actually, if we can factorize the joint cdf or joint pdf or joint pmf as the product of some functions in each of the variables, then the condition is also satisfied.

Example. The joint pdf of two independent exponential random variables with rate , and is (Random variables and are said to be independent and identically distributed (i.i.d.) in this case)

In general, the joint pdf of independent exponential random variables with rate , is (Random variables are also i.i.d. in this case)

On the other hand, if the joint pdf of two random variables and are random variables and are dependent since the joint pdf cannot be factorized as the product of marginal pdf's.

Clipboard

Exercise. Let be jointly continuous random variables. Consider a joint pdf of :

1 Calculate .

1
2
3
4

2 Are independent?

yes
no


Consider another joint pdf of :

1 Calculate .

1
2
3
4

2 Are independent?

yes
no


Consider another joint pdf of :

1 Calculate .

1
2
3
4

2 Are independent?

yes
no


Proposition. (Independence of events concerning disjoint sets of independent random variables) Suppose random variables are independent. Then, for each and fixed functions , the random variables are independent.

Example. Suppose are independent Bernoulli random variables with success probability . Then, and are also independent.

On the other hand, and are not independent. A counter-example to the independence is Left hand side equals zero since , but .

Right hand side may not equal zero since , and . We can see that may not equal zero.

Clipboard

Exercise.

Let be i.i.d. random variables, and also be i.i.d. random variables. Which of the following is (are) true?

and are independent.
and are independent.
and are independent.
and are independent if are independent.


Sum of independent random variables (optional)

[edit | edit source]

In general, we use joint cdf, pdf or pmf to determine the distribution of sum of independent random variables by first principle. In particular, there are some interesting results related to the distribution of sum of independent random variables.

Sum of independent random variables

Proposition. (Convolution of cdf's and pdf's) If the cdf of independent random variables and are and respectively, then the cdf of is and the pdf of is

Proof.

  • Continuous case:
  • cdf:
/\                                     
//\ y                                
///\|
////*
////|\
////|/\
////|//\ x+y=z <=> x=z-y
////|///\
////|////\
----*-----*--------------- x 
////|//////\
////|///////\

-->: -infty to z-y
^
|: -infty to infty
 
*--*
|//| : x+y <= z
*--*
  • pdf:

Remark.

  • The cdf and pdf in this case are actually the convolution of the cdf's and , and pdf's (pmf's) and respectively, and hence the name of the proposition.

Example.

  • Let the pdf of be .
  • Let the pdf of be .
  • Then, the pdf of is

Graphically, the pdf looks like

        y
        |
        |
        |
     *  * 1
      \ |\  
  y=-z \| \ y=1-z
-----*--*--*----- z
    -1 O|  1   
        |
     -1 *
        |
Clipboard

Exercise.

1 Calculate .

0
1/4
1/2
3/4
1

2 Calculate .

0
1/4
1/2
3/4
1

3 Calculate such that .

-1/2
-1/4
0
1/4
1/2



Proposition. (Convolution of pmf's) If the pmf of independent random variables and are and respectively, then the pmf of is

Proof.

  • Let .
  • For each nonnegative integer ,

  • Since for each , 's are pairwise disjoint.
  • Hence, by extended P3 and independence of and ,
  • The result follows by definition.

Example. We roll a fair six-faced dice twice (independently). Then, the probability for the sum of the numbers coming up to be 7 is .

Proof. Let and be the first and second number coming up respectively. The desired probability is

Clipboard

Exercise.

1 Calculate the probability for the sum to be 6 instead.

1/12
1/6
5/36
7/36
4/9

2 The probability for the sum to be is 0. Which of the following is (are) possible value(s) of ?

1
2
3
12
13

3 Suppose the dice is loaded such that the probability for the number coming up to be 6 is now , and for other numbers, they are equally likely to be coming up. Calculate the probability for the sum to be 7 now.

0.1
0.101
0.1001
0.10000001
0.167



Proposition. (Sum of independent Poisson r.v.'s) If and are independent, then .

Proof.

  • The pmf of is

  • This pmf as the pmf of , and so .
  • We can extend this result to Poisson r.v.'s by induction.

Example. There are two service counters, for which the first one receives enquiries per hour, while the second one receives enquiries per hour. Given that and are independent, the number of enquiries received by the two counters per hour follows .

Proof.

  • The number of enquiries received by the two counters per hour is .
  • Then, the result follows from the proposition about sum of Poisson r.v.'s.

Clipboard

Exercise.

Which distribution does the number of enquiries received by the first counter for two hours follow?



Order statistics

[edit | edit source]

Definition. (Order statistics) Let be i.i.d. r.v.'s (each with cdf ). Define be the smallest, second smallest, ..., largest of . Then, the ordered values is the order statistics.

Proposition. (Cdf of order statistics) The cdf of ( is an integer such that ) is

Proof.

  • Consider the event .
                          Possible positions of x
                      |<--------------------->
    *---*----...------*----*------...--------*
X  (1)  (2)          (k)  (k+1)             (n)
                      |----------------------> when x moves RHS like this, >=k X_i are at the LHS of x
  • We can see from the above figure that .
  • Let no. of 's that are less than or equal to be .
  • Since (because for each , we can treat and be the two outcomes in a Bernoulli trial),
  • The cdf is

Example. Let be i.i.d. r.v.'s following . Then, the cdf of is

Clipboard

Exercise.

Calculate .

0.000665
0.000994
0.036296
0.963704
0.999335




Poisson process

[edit | edit source]

Definition.

Illustration of Poisson process. Each circle indicates one arrival. The arrivals occur at common rate , and the successive interarrivial times are independent.

If successive interarrival times of unpredictable events are independent random variables, with each following an exponential distribution with a common rate , then the process of arrivals is a Poisson process with rate .

There are several important properties for Poisson process.

Proposition. (Time to -th event in Poisson process) The time to -th event in a Poisson process follows the distribution.

Proof.

  • The time to -th event is , with each following .
  • It suffices to prove that , and then the desired result follows by induction.
which is the pdf of , as desired.

Remark. The time to -th event is also the sum of the successive interarrival times before the -th event.

Proposition. (Number of arrivals within a fixed time interval) The number of arrivals within a fixed time interval of length follows the distribution.

Proof. For each nonnegative integer , let be the interarrival time between the -th and -th arrival, and be the time to th event, starting from the beginning of the fixed time interval (we can treat the start to be time zero because of the memoryless property). The joint pdf of is Let the number of arrivals within the fixed time interval. The pmf of is which is the pmf of . The result follows.

Proposition. (Time to the first arrival with independent Poisson processes) Let be independent random variables with , in which . If we define (which is the time to the first arrival with independent Poisson processes), then .

Proof. For each ,

Example. Suppose there are two service counters, counter A and B, with independent service times following the exponential distribution with rate . In the past 10 minutes, John and Peter are being served at counter A and B respectively.

First, the time you need to wait to be served (i.e. the time for one of John and Peter leaves the counter) is the minimum value of the service time for John and Peter counting from now, which are independent and follow the exponential distribution with rate . Thus, your waiting time follows the exponential distribution with rate .

Suppose now John leaves the counter A, and you are currently being served at counter A. Then, the probability that you leave the counter first is , by memoryless property and symmetry (the chances that Peter and you leave the counter first are governed by the same chance mechanism), counterintuitively.

Clipboard

Exercise. Suppose the process of arrivals of car accidents is a Poisson process with unit rate. Let be the time to the -th car accidents, and be the interarrival time between the -th and -th accidents.

1 Which of the following is (are) true?

2 Which of the following is (are) true?

3 Which of the following is (are) true?

The pmf of the number of arrivals within a fixed time interval of length is .