Probability

Important definitions of probability can be discussed under the following heads.

1. Probability as a measure of belief (Subjective)

2. Classical definition of probability (Mathematical)

3. Frequency definition of probability (Statistical)

4. Axiomatic definition of probability (Modern)

Subjective Probability
Subjective probability is a type of probability derived from an individual’s
personal judgment or own experience about whether a specific outcome is likely
to occur. It contains no formal calculations and only reflects the subject’s
opinions and past experience.

Classical Definition of Probability (Mathemati-

cal or ‘a priori’)
Classical definition is the oldest and simplest definition of probability. This
is sometimes called equally -likely events approach. It is also known by the
name Laplace definition. From a practical point of view it is the most useful
definition of probability.

Basic Concepts
Random Experiment
An experiment is a process at the completion of which we observe some results.
The experiments can be either deterministic or random. The experiments which
have only one possible outcome are called deterministic experiments. An ex-
periment is said to be random if

1. It can be repeated under identical conditions

2. It has two or more possible outcomes.

3. All the possible outcomes are known in advance.

4. It is not possible to predict the outcome of a particular trial of the ex-

periment with certainty.
 2

Examples

1. Tossing a coin

2. Rolling a die

3. Weight of a newborn baby

4. Weather conditions of a region

Trial
An attempt to produce an outcome of a random experiment is called a trial.
Examples If we toss a coin or roll a die, we are performing a trial.

Outcome
The result of a random experiment is called an outcome.

Sample Space
Set of all possible outcomes of a random experiment is called a sample space.
It is demoted by S or Ω.
Example

1. In the tossing of a coin, the sample space is Ω = {H, T }.

2. While rolling a die, Ω = {1, 2, 3, 4, 5, 6}

Event
An event is a subset of a sample space. That is, of all possible outcomes of a
random experiment, some outcomes satisfies a specific description, which we
call an event. They are denoted by capital letters A, B, C etc.
Example In the experiment of rolling a die, if A be the event of getting a an
odd face, then A = {1, 3, 5}. If B is the event of getting a face whose face value
is less than 3, then B = {1, 2}. If C is the event of getting the face 5, then
C = {5}.

Simple or Elementary Event

Events which contains only a single outcome are called elementary events.
Example In the experiment of rolling a die, A = {5}, the event of getting the
face 5, is an elementary event.
 3

Equally Likely Events

Events are said to be equally likely when we have no reason to expect one
rather than the other.
Example In tossing an unbiased coin, the two events head and tail are equally
likely, since we have no reason to expect head rather than tail or tail rather
than head.

Exhaustive Events
Set of all possible outcomes in a trial constitutes the set of exhaustive cases.
When two or more events collectively form the sample space, than it is known
as collectively exhaustive events.
Example In the experiment of rolling a die, if A be the event of getting an
odd face and B the event of getting an even face. then A and B are collectively
exhaustive. That is, A ∪ B = Ω.

Mutually Exclusive Events

Events are said to be mutually exclusive or incompatible or disjoint if the
happening of any one of them excludes the happening of all the others in a
trial. That is, no two or more of them can happen simultaneously in the same
trial. Hence, in the case of mutually exclusive events, their intersection will be
the null set.
Example The events of turning a head or tail in tossing a coin are mutually
exclusive. In the rolling of a die all the six faces are mutually exclusive. In this
same experiment, if A = {1, 2}, B = {3}, C = {5, 6}, then A, B and C are
mutually exclusive.

Favourable Cases
The outcomes which aid for the occurrence of an event are called its favorable
cases.
Example In the rolling of a die if A = {1, 2}, the turning up of the faces 1 or
2 are the cases favorable for the event A.

Definition 1. If a trial of a random experiment results in n mutually exclusive,

equally likely and exhaustive outcomes and m (m ≤ n) are favorable to the
occurrence of an event A, then probability of A denoted by P (A) is given by

m Number of favorable cases

P (A) = = .
n Total number of cases

Obviously, 0 ≤ P (A) ≤ 1.
 4

Remark 1. When P (A) = 0, A is called an impossible event.

When P (A) = 1, A is called a sure event.
When 0 < P (A) < 1, A is called a random event.

Remark 2. We can represent the probability of A by saying that the odds in

favor of A as m : (n − m) or the odds against A as (n − m) : m.

Limitations of classical definition

1. The above definition of mathematical probability fails in the following
cases. In the classical or a priori definition of probability only equally
likely cases are taken into consideration. If the events cannot be consid-
ered equally likely classical definition fails to give a good account of the
concept of probability.

2. When the total number of possible outcomes ’n’ be- come infinite or
countably infinite, this definition fails to give a measure for probability.

3. If we are deviating from the games of chances like, tossing a coin, throwing
a die etc. this definition cannot be applied.

4. Another limitation is that it does not contribute much to the growth of

the probability theory.

Frequency Definition of Probability

Statistical Regularity
Let there be a random experiment and A be an event. Denote ‘m’ as the
number of occurrences of the event A in ‘n’ identical trials. Then the ratio
m/n is called ‘frequency ratio’ or relative frequency of the event A. If we
increase the number of trials, then the ratio m/n shows some sort of regularity.
That means, m/n will approximate to the value p = P (A) as n increases and
this approximation will improve as n increases further. This tendency of the
frequency ratio to approach a definite value is called the notion of statistical
regularity.
An example to illustrate the stabilisation of frequency ratios.

No. of tosses 10 25 50 100 200 400 800 1000 10000

No. of heads 6 10 28 54 96 203 398 498 4990
frequency ratio 0.60 0.40 0.56 0.54 0.48 0.5075 0.4795 0.498 0.499

The frequency ratio fluctuates violently for small values of n, but gradu-
ally the amplitude of the fluctuations become smaller and smaller and would
 5

approach some definite, value say ‘p’, the probability of the event A. This
‘stability of frequency ratios’ usually appears in long series of repeated random
observations performed under uniform No. of heads conditions.
Definition 2. Let the trials be repeated over a large number of times under
essentially homogeneous conditions. The limit of the ratio of the number of
times an event A happens (m) to the total number of trials (n), as the number
of trials tends to infinity is called the probability of the event A, That is,
m
P (A) = lim .
n→∞ n

It is, however, assumed that the limit is unique as well as finite.

Remark 3. The application of this definition to many problems cannot be
extensive since n is usually finite and the limit of the ratio cannot normally be
taken as it leads to mathematical difficulties. Besides, the definition of prob-
ability thus proposed by Von Mises would involve a mixture of empirical and
theoretical concepts, which is usually avoided in modern axiomatic approach.
Remark 4. The two definitions of probability are apparently different. The
mathematical definition is the relative frequency of favorable cases to the total
number of cases while in the statistical definition it is the limit of the relative
frequency of the happening of the event.

Theorems in Probability
Theorem 1. Probability of an impossible event is zero. That is,
P (φ) = 0.
Theorem 2. Probability of a sure event is one. That is,
P (Ω) = 1.
Theorem 3. If A ⊂ B, then
P (A) ≤ P (B).
Theorem 4. For any event A,
P (Ac ) = 1 − P (A).
Theorem 5. If A1 , A2 , . . . An are disjoint events, then probability of
P (A1 ∪ A2 ∪ . . . ∪ An ) = P (A1 ) + P (A2 ) + . . . + P (An ).
Theorem 6. If A and B are any two events,
P (A ∪ B) = P (A) + P (B) − P (A ∩ B).
Theorem 7. If A, B and C are any three events,
P (A∪B∪C) = P (A)+P (B)+P (C)−P (A∩B)−P (B∩C)−P (A∩C)+P (A∩B∩C).
 6

Conditional Probability
We have defined the probability for the occurrence of an event A relative to the
outcome set Ω or S. In other words, when an event is considered, already an
outcome set Ω of which A is a subset is specified. Hence it is more appropriate to
call the probability of A as the probability of A given Ω. This may be written
as P (A|Ω). Every probability statement we made so far was a conditional
statement of this nature.
Now we will consider a different type of conditional statement such as the
probability of an event A given another event B has occurred, where B need
not be sure event or the outcome set Ω. The method of defining a conditional
probability P (A|B), the probability of A given that the event B has already
occurred, can be seen from the following illustration.
Suppose a population of N people contains N1 graduates and N2 males.
Let A be the event that ‘a person chosen at random is a graduate’ and B be
the event that ‘a person chosen at random is a male?
Then P (A) = N1 /N , P (B) = N2 /N .

Instead of the entire population we may investigate the male sub population
and require the probability that “a male chosen at random is a graduate”. Let
the number of males who are graduates be N3 , which can be represented by the
event A ∩ B and we have P (A ∩ B) = N3 /N . Then the probability that a male
chosen at random is a graduate = N3 /N2 . This is a conditional probability
denoted by P (A|B). That is,

N3
P (A|B) =
N2
N3 /N
=
N2 /N
P (A ∩ B)
= .
P (B)
 7

Definition 3. Let A and B be any two events. The probability of the event A
given that the event B has already occurred or the conditional probability of A
given B, denoted by P (A|B), is defined as
P (A ∩ B)
P (A|B) = , P (B) 6= 0.
P (B)
Similarly the conditional probability of B given A is defined as
P (A ∩ B)
P (B|A) = , P (A) 6= 0.
P (A)
Remark 5. For P (B) > 0, P (A|B) ≤ P (A).
Remark 6. P (A|B) is not defined if P (B) = 0.
Remark 7. P (B|B) = 1.

Multiplication Law of Probability

For any two events A and B,
P (A ∩ B) = P (A) P (B|A), P (A) > 0
= P (B) P (A|B), P (B) > 0.

Independent Events
Definition 4. Two or more events are said to be independent if the probability
of any one of them is not affected by the supplementary knowledge concerning
the materialisation of any number of the remaining events. Otherwise they are
said to be dependent.

Independence of two events A and B

An event A is said to be independent (statistically independent) of event B, if
the conditional probability of A given B, is equal to the unconditional proba-
bility of A. In symbols,
P (A|B) = P (A).
Similarly, if the event B is independent of A, we must have
P (B|A) = P (B).
Since, P (A ∩ B) = P (A)P (B|A) and since P (B|A) = P (B) when B is inde-
pendent of A, we must have
P (A ∩ B) = P (A) P (B).
 8

Hence, the events A and B are independent if

P (A ∩ B) = P (A) P (B).

Pairwise Independent Events

Definition 5. Three events A, B and C are said to be pairwise independent if
P (A ∩ B) = P (A) P (B)
P (B ∩ C) = P (B) P (C)
P (A ∩ C) = P (A) P (C).

Mutually Independent Events

Definition 6. Three events A, B and C are said to be mutually independent
if
P (A ∩ B) = P (A) P (B)
P (B ∩ C) = P (B) P (C)
P (A ∩ C) = P (A) P (C)
P (A ∩ B ∩ C) = P (A) P (B) P (C).
Remark 8. Pairwise independence does not implies mutual independence.

Multiplication Theorem of Independent Events

For any two independent events A and B,
P (A ∩ B) = P (A) P (B).

Addition Theorem of Independent Events

For any two independent events A and B,
P (A ∪ B) = P (A) + P (B) − P (A) P (B).
Theorem 8. For any two independent events A and B,
1. A and B 0 are independent.
2. A0 and B are independent.
3. A0 and B 0 are independent.
Remark 9. If A and B are independent and mutually exclusive events then
either P (A) = 0 or P (B) = 0.
 9

Total Probability Theorem and Baye’s Theorem

Theorem 9 (Total Probability Theorem). Let B1 , B2 , . . . , Bn be a partition

of the sample space Ω into n mutually exclusive events such that P (Bi ) > 0 for
i = 1, 2, . . . , n. Then for any arbitrary event A
n
X
P (A) = P (Bi )P (A|Bi ).
i=1

Theorem 10 (Baye’s Theorem). Let Ω be a sample space partitioned

into n mutually exclusive events B1 , B2 , . . . , Bn such that P (Bi ) > 0 for
i = 1, 2, . . . , n. Let A be any event of Ω for which P (A) 6= 0. Then the
probability for the event Bi , (i = 1, 2...n) given the event A is
P (Bi )P (A|Bi )
P (Bi |A) = Pn .
i=1
P (Bi )P (A|Bi )
Note :
Posteriori probability - P (Bi |A), i = 1, 2...n, the probabilities determined
after observing the event A, are called posteriori probabilities.
Priori Probability - P (Bi ), i = 1, 2...n, the probabilities determined before
observing the event A, are called priori probabilities.

Axiomatic Definition of Probability

Class of Sets
A set whose elements are again sets is called a class of sets.

Power Set of A
The set of all subsets of A is called the power set of A, denoted by P(A). If A
is a given set then P(A) is a class in A. If A has n elements, then P(A) has 2n
elements in it.
 10

σ- field in A
A non-empty class F of subsets of A is called a σ- field in A if it is closed
under complementation and countable union or intersection. i.e., if

1. B c ∈ F , ∀ B ∈ F

2. ∞ Bi ∈ F ∀ Bi ∈ F , i ≥ 1.
S
i=1

Borel σ- field
σ- field over the real line R is called the Borel σ- field denoted by B. Hence
elements of B are subsets of R.

Measurable Space
Let A be an arbitrary set and F a σ-field in A. Then the pair (A, F ) is said
to be a measurable space.

Sample Space
Set of all possible outcomes of a random experiment is denoted by Ω. σ- field in
Ω, denoted by A , is a class of events. i.e., elements of A are events of Ω. The
pair (Ω, A ) called the sample space of a random or statistical experiments.

Set Function
A set function η, is a function from a class of sets A to the extended real line
R̄ (R̄ = R ∪ {−∞, ∞}). Denoted by η : A → R̄. That is, the domain of a set
function will be a class (i.e., its elements will be set) and codomain will be the
real line.

Measure
Let (A, F ) be a measurable space. A set function µ, defined on F is called a
measure if

1. µ(B) ≥ 0 ∀B ∈ F

2. µ(φ) = 0

3. If {Bi } is a sequence of disjoint sets in F , then µ( ∞

S P∞
i=1
Bi
) = i=1
µ(Bi ).

Denoted by, µ : F → R̄. i.e., A non-negative countably additive set function

µ defined on the σ-field is a measure.
 11

Probability - Axiomatic Definition

Let (Ω, A ) be a measurable space. A set function P , defined on A is called a
probability measure if

1. 0 ≤ P (A) ≤ 1 ∀A ∈ A

2. P (φ) = 0 and P (Ω) = 1

3. If {Ai } is a disjoint sequence of sets in A , then P ( ∞ Ai ) = ∞

S P
i=1 i=1
P (Ai ).

The condition P (Ω) = 1 makes the probability measure a normed measure.

Measure Space
Let (A, F ) be a measurable space and µ a measure on F . Then the triplet
(A, F , µ) is known as a measure space.
In particular, if µ is the probability measure P , the triplet (Ω, A , P ) is called
the probability space.

Measurable Function
Let (A, F ) be a measurable space and f : A → R be a real valued function
defined on A. Then f is a measurable function on (A, F ) if

f −1 (B) = {a ∈ A|f (a) ∈ B} ∈ F , ∀B ∈ B,

the Borel σ- field.

Borel Measurable Function

If a real valued function f : R → R is such that f −1 (−∞, x) ∈ B, for all x ∈ R,
then f is said to be a Borel measurable function.