PROBABILITY DISTRIBUTIONS: DISCRETE AND CONTINUOUS

Univariate Probability Distributions. Let S be a sample space with a prob-

ability measure P defined over it, and let x be a real scalar-valued set function
defined over S, which is described as a random variable.
If x assumes only a finite number of values in the interval [a, b], then it is said
to be discrete in that interval. If x assumes an infinity of values between any two
points in the interval, then it is said to be everywhere continuous in that interval.
Typically, we assume that x is discrete or continuous over its entire domain, and
we describe it as discrete or continuous without qualification.
If x ∈ {x1 , x2 , . . .}, is discrete, then a function f (xi ) giving the probability
that x = xi is called a probability mass function. Such a function must have the
properties that


f (xi ) ≥ 0, for all i, and f (xi ) = 1.
i

Example. Consider x ∈ {0, 1, 2, 3, . . .} with f (x) = (1/2)x+1 . It is certainly true

that f (xi ) ≥ 0 for all i. Also,

 
1 1 1 1
f (xi ) = + + + + · · · = 1.
i
2 4 8 16

To see this, we may recall that

1
= {1 + θ + θ2 + θ3 + · · ·},
1−θ

whence
θ
= {θ + θ2 + θ3 + +θ4 + · · ·}.
1−θ
Setting θ = 1/2 in the expression above gives θ/(1 − θ) = 12 /(1 − 12 ) = 1, which is
the result that we are seeking.
If x is continuous, then a probability density function (p.d.f.) f (x) may be defined
such that the probability of the event a < x ≤ b is given by
 b
P (a < x ≤ b) = f (x)dx.
a

Notice that, when b = a, there is

 a
P (x = a) = f (x)dx = 0.
a

That is to say, the integral of the continuous function f (x) at a point is zero. This
corresponds to the notion that the probability that the continuous random variable

1
x assumes a particular value amongst the non denumerable (i.e. uncountable) in-
finity of values within its domain must be zero. Nevertheless, the value of f (x) at
a point continues to be of interest; and we describe this as a probability measure
as opposed to a probability.
Notice also that it makes no difference whether we include of exclude the
endpoints of the interval [a, b] = {a ≤ x ≤ b}. The reason that we have chosen to
use the half-open half-closed interval (a, b] = {a < x ≤ b} is that this accords best
with the definition of a definite integral over [a, b], which is obtained by subtracting
the integral over (−∞, a] from the integral over (−∞, b].
Example. Consider the exponential function

1 −x/a
f (x) = e
a
defined over the interval [0, ∞) and with α > 0. There is f (x) > 0 and
 ∞  ∞
1 −x/a −x/a
e dx = −e = 1.
0 a 0

Therefore, this function constitutes a valid p.d.f. The exponential distribution

provides a model for the lifespan of an electronic component, such as fuse, for
which the probability of failing in the ensuing period is liable to be independent of
how long it has survived already.
Corresponding to any p.d.f f (x), there is a cumulative distribution function,
denoted by F (x), which, for any value x∗ , gives the probability of the event x ≤ x∗ .
Thus , if f (x) is the p.d.f. of x, which we denote by writing x ∼ f (x), then
 x∗
∗
F (x ) = f (x)dx = P (−∞ < x ≤ x∗ ).
−∞

In the case where x is a discrete random variable with a probability mass function
f (x), also denoted by x ∼ f (x), there is


F (x∗ ) = f (x) = P (−∞ < x ≤ x∗ ).
x≤x∗

Pascal’s Triangle and the Binomial Expansion. The binomial distribution is

one of the fundamental distributions of mathematical statistics. To derive the bino-
mial probability mass function from first principles, it is necessary to invoke some
fundamental principles of combinatorial calculus. We shall develop the necessary
results at some length in the ensuing sections before deriving the mass function.
Consider the following binomial expansions:

(p + q)0 = 1,
(p + q)1 = p + q,
(p + q)2 = p2 + 2pq + q 2 ,

2
 (p + q)3 = p3 + 3p2 q + 3pq 2 + q 3 ,

(p + q)4 = p4 + 4p3 q + 6p2 q 2 + 4pq 3 + q 4 ,

(p + q)5 = p5 + 5p4 q + 10p3 q 2 + 10p2 q 3 + 5pq 4 + q 5 .

The generic expansion is in the form of

n(n − 1) n−2 2 n(n − 1)(n − 2) n−3 3

(p + q)n =pn + npn−1 q + p q + p q +
2 3!
n(n − 1) · · · (n − r + 1) n−r r
··· + p q + ···
r!
n(n − 1)(n − 2) 3 n−3 n(n − 1) 2 n−2
+ p q + p q + npq n−1 + pn .
3! 2

In a tidier notation, this becomes

n
n!
n
(p + q) = px q n−x .
x=0
(n − x)!x!

We can find the coefficient of the binomial expansions of successive degrees by

the simple device known as Pascal’s triangle:

1 1

1 2 1

1 3 3 1

1 4 6 4 1

1 5 10 10 5 1
The numbers in each row but the first are obtained by adding two adjacent numbers
in the row above. The rule is true even for the units that border the triangle if we
suppose that there are some invisible zeros extending indefinitely on either side of
each row.
Instead of relying merely upon observation to establish the formula for the
binomial expansion, we should prefer to derive the formula by algebraic methods.
Before we do so, we must reaffirm some notions concerning permutations and com-
binations that are essential to a proper derivation.

Permutations. Let us consider a set of three letters {a, b, c} and let us find the
number of ways in which they can be can arranged in a distinct order. We may
pick any one of the three to put in the first position. Either of the two remaining
letters may be placed in the second position. The third position must be filled by
the unused letter. With three ways of filling the first place, two of filling the second

3
and only one way of filling the third, there are altogether 3 × 2 × 1 = 6 different
arrangements. These arrangements or permutations are

abc, cab, bca, cba, bac acb.

Now let us consider an unordered set of n objects denoted by

{xi ; i = 1, . . . , n},

and let us ascertain how many different permutations arise in this case. The answer
can be found through a litany of questions and answers which we may denote by
[(Qi , Ai ); i = 1, . . . , n]:
Q1 : In how may ways can the first place be filled? A1 : n
Q2 : In how may ways can the second place be filled? A2 : n − 1 ways,
Q3 : In how may ways can the third place be filled? A3 : n − 2 ways,
..
.
Qr : In how may ways can the rth place be filled? Ar : n − r + 1 ways,
..
.
Qn : In how may ways can the nth place be filled? An : 1 way.
If the concern is to distinguish all possible orderings, then we can recognise

n(n − 1)(n − 2) · · · 3.2.1 = n!

different permutations of the objects. We call this number n-factorial, which is

written as n!, and it represents the number of permutations of n objects taken n
at a time. We also denote this by
n
Pn = n!.

We shall use the same question-and-answer approach in deriving several other im-
portant results concerning permutations. First we shall ask
Q: How many ordered sets can we recognise if r of the n objects are so alike as to
be indistinguishable?
A reasoned answer is as follows:
A: Within any permutation there are r objects that are indistinguishable. We can
permute the r objects amongst themselves without noticing any differences.
Thus the suggestion that there might be n! permutations would overestimate
the number of distinguishable permutations by a factor of r!; and, therefore,
there are only
n!
recognizably distinct permutations.
r!
Q: How many distinct permutations can we recognise if the n objects are divided
into two sets of r and n − r objects—eg. red billiard balls and white billiard
balls—where two objects in the same set are indistinguishable?

4
 A: By extending the previous argument, we should find that the answer is

n!
.
(n − r)!r!

Q: How many ways can we construct a permutation of r objects selected from a

set of n objects?
A: There are two ways of reaching the answer to this question. The first is to
use the litany of questions and answers which enabled us to discover the total
number of permutations of n objects. This time, however, we proceed no
further than the question Qr , for the reason that there are no more than r
places to fill. The answer we seek is the number of way of filling the first r
places. The second way is to consider the number of distinct permutation of
n objects when n − r of them are indistinguishable. We fail to distinguish
amongst these objects because they all share the same characteristic which is
that they have been omitted from the selection. Either way, we conclude that
the number is
n
Pr = n(n − 1)(n − 2) · · · (n − r + 1)
n!
=
(n − r)!

Combinations. Combinations are selections of objects in which no attention is

payed to the ordering. The essential result is found in answer to just one question:
Q: How many ways can we construct an unordered set of r objects selected from
amongst n objects?
A: Consider the total number of permutations of r objects selected from amongst
n. This is n Pr . But each of the permutations distinguishes the order of the
r objects it comprises. There are r! different orderings or permutations of r
objects; and, to find the number of different selections or combinations when
no attention is paid to the ordering, we must deflate the number n Pr by a
factor of r!. Thus the total number of combinations is

n 1n n!
Cr = Pr =
r! (n − r)!r!
= n Cn−r .

Notice that we have already derived precisely this number in answer to a seemingly
different question concerning the number of recognizably distinct permutations of
a set of n objects of which r were in one category and n − r in another. In the
present case, there are also two such categories: the category of those objects which
are included in the selection and the category of those which are excluded from it.

Example. Given a set of n objects, we may define a so-called power set, which
is the set of all sets derived by making selections or combinations of these objects.
It is straightforward to deduce that there are exactly 2n objects in the power set.

5
This is demonstrated by setting p = q = 1 in the binomial expansion above to reach

n

the conclusion that
n n!
2 = = .
x
x x
x!(n − x)!

Each element of this sum is the number of ways of selcting x objects from amongst
n, and the sum is for all values of x = 0, 1, . . . , n.

The Binomial Theorem. Now we are in a position to derive our conclusion

regarding the binomial theorem without, this time, having recourse to empirical
induction. Our object is to determine the coefficient associated with the generic
term px q n−x in the expansion of

(p + q)n = (p + q)(p + q) · · · (p + q),

where the RHS displays the n factors that are to be multiplied together. The
coefficients of the various elements of the expansion are as follows:
pn The coefficient is unity, since there is only one way of choosing n of the
p’s from amongst the n factors.
pn−1 q This term is the product of q selected from one of the factors and n − 1 p’s
provided by the remaining factors. There are n = n C1 ways of selecting
the single q.
pn−2 q 2 The coefficent associated with this term is the number of ways of selecting
two q’s from n factors, which is n C2 = n(n − 1)/2.
..
.
pn−r q r The coefficent associated with this terms is the number of ways of selecting
r q’s from n factors, which is n Cr = n!/{(n − r)!r!}.
From such reasoning, it follows that

(p + q)n =pn + n C1 pn−1 q + n C2 pn−2 q 2 +

· · · + n Cr pn−r q r + · · ·
+ n Cn−2 p2 q n−2 + n Cn−1 pq n−1 + q n

n
n!
= px q n−x .
x=0
(n − x)!x!

The Binomial Probability Distribution. We wish to find, for example, the

number of ways of getting a total of x heads in n tosses of a coin. First we consider
a single toss of the coin. Let us take the ith toss, and let us denote the outcome by
xi = 1 if it is heads and by xi = 0 if it is tails. Heads might be described as a sucess,
whence the probability of a sucess will be P (xi = 1) = p. Tails might be described
as a failure and the corresponding probability of this event is P (xi = 0) = 1 − p.
For a fair coin, we should have p = 1 − p = 12 , of course.

6
 We are now able to define a probabilty function for the outcome of the ith
trial. This is
f (xi ) = pxi (1 − p)1−xi with xi ∈ {0, 1}.
The experiment of tossing a coin once is called a Bernoulli trial, in common with
any other experiment with a random dichotomous outcome. The corresponding
probability function is called a point binomial.
If the coin is tossed n times, then the probability of any particular sequence of
heads and tails, denoted by (x1 , x2 , . . . , xn ) is given by

P (x1 , x2 , . . . , xn ) = f (x1 )f (x2 ) · · · f (xn )

=p xi
(1 − p) n− xi

= px (1 − p)n−x ,

where we have defined x = xi . This result follows from the independence of
the Bernouilli trials, whereby P (xi , xj ) = P (xi )P (xj ) is the probability of the
occurrence of xi and xj together in the sequence.
Altogether, there are
 
n n!
= = nCx
x (n − x)!x!

different sequences (x1 , x2 , . . . , xn ) which contain x heads; so the probability of the

event of x heads in n tosses is given by

n!
b(x; n, p) = px q n−x .
(n − x)!x!

This is called the binomial probability mass function.