Outline

Our goal
Randomized Algorithms – Two types (Two examples)
I Monte Carlo

I Las Vegas

Basics of randomization – probability space, expectation, events,

random variables, etc.

Analysis for two examples

Fahad (IITH) CS5390 2 2 / 58

 Part I

Randomized Algorithms: Two

Examples

Fahad (IITH) CS5390 3 3 / 58

Randomized Algorithms

Input x Output y
Deterministic Algorithm

Fahad (IITH) CS5390 4 4 / 58

Randomized Algorithms

Input x Output y
Deterministic Algorithm

random bits r

Input x Output yr
Randomized Algorithm

Fahad (IITH) CS5390 4 4 / 58

Randomized Algorithms

Input x Output y
Deterministic Algorithm

random bits r

Input x Output yr
Randomized Algorithm

The behaviour of the algorithm is “not always good”.

Fahad (IITH) CS5390 4 4 / 58

Randomized Algorithms

Input x Output y
Deterministic Algorithm

random bits r

Input x Output yr
Randomized Algorithm

The behaviour of the algorithm is “not always good”.

Output may not be correct always (Monte Carlo)

Fahad (IITH) CS5390 4 4 / 58

Randomized Algorithms

Input x Output y
Deterministic Algorithm

random bits r

Input x Output yr
Randomized Algorithm

The behaviour of the algorithm is “not always good”.

Output may not be correct always (Monte Carlo)
Running time may not be fast always (Las Vegas)
Fahad (IITH) CS5390 4 4 / 58
Example 1: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Fahad (IITH) CS5390 5 5 / 58

Example 1: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Deterministic algorithm:
1 Multiply A and B and check if equal to C .
2 Running time?

Fahad (IITH) CS5390 5 5 / 58

Example 1: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Deterministic algorithm:
1 Multiply A and B and check if equal to C .
2 Running time? O(n3 ) by straight forward approach. O(n2.37 )
with fast matrix multiplication (complicated and impractical).

Fahad (IITH) CS5390 5 5 / 58

Example: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Fahad (IITH) CS5390 6 6 / 58

Example: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Randomized algorithm:
1 Pick a random n ⇥ 1 vector r 2 {0, 1}n .
2 Return the answer of the equality ABr = Cr .
3 Running time?

Fahad (IITH) CS5390 6 6 / 58

Example: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Randomized algorithm:
1 Pick a random n ⇥ 1 vector r 2 {0, 1}n .
2 Return the answer of the equality ABr = Cr .
3 Running time? O(n2 )!

Fahad (IITH) CS5390 6 6 / 58

Example: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Randomized algorithm:
1 Pick a random n ⇥ 1 vector r 2 {0, 1}n .
2 Return the answer of the equality ABr = Cr .
3 Running time? O(n2 )!
Theorem
If AB = C then the algorithm will always say YES. If AB 6= C then
the algorithm will say YES with probability at most 1/2. Can repeat
the algorithm 100 times independently to reduce the probability of a
false positive to 1/2100 .
Fahad (IITH) CS5390 6 6 / 58
Example 2: Randomized QuickSort
QuickSort ?
1 Pick a pivot element from array
2 Split array into 3 subarrays: those smaller than pivot, those
larger than pivot, and the pivot itself.
3 Recursively sort the subarrays, and concatenate them.

Randomized QuickSort
1 Pick a pivot element uniformly at random from the array
2 Split array into 3 subarrays: those smaller than pivot, those
larger than pivot, and the pivot itself.
3 Recursively sort the subarrays, and concatenate them.

Fahad (IITH) CS5390 7 7 / 58

Example 2: Randomized Quicksort
Recall: QuickSort can take ⌦(n2 ) time to sort array of size n.

Fahad (IITH) CS5390 8 8 / 58

Example 2: Randomized Quicksort
Recall: QuickSort can take ⌦(n2 ) time to sort array of size n.

Theorem
Randomized QuickSort sorts a given array of length n in
O(n log n) expected time.

Fahad (IITH) CS5390 8 8 / 58

Example 2: Randomized Quicksort
Recall: QuickSort can take ⌦(n2 ) time to sort array of size n.

Theorem
Randomized QuickSort sorts a given array of length n in
O(n log n) expected time.

Note: On every input randomized QuickSort takes O(n log n) time

in expectation. On every input it may take ⌦(n2 ) time with some
small probability.

Fahad (IITH) CS5390 8 8 / 58

Why randomized algorithms?
1 Many many applications in algorithms, data structures and
computer science!
2 In some cases only known algorithms are randomized or
randomness is provably necessary.
3 Often randomized algorithms are (much) simpler and/or more
efficient.
4 Several deep connections to mathematics, physics etc.
5 ...
6 Lots of fun!

Fahad (IITH) CS5390 9 9 / 58

Average case analysis vs Randomized algorithms
Average case analysis:
1 Fix a deterministic algorithm.
2 Assume inputs comes from a probability distribution.
3 Analyze the algorithm’s average performance over the
distribution over inputs.

Randomized algorithms:
1 Algorithm uses random bits in addition to input.
2 Analyze algorithms average performance over the given input
where the average is over the random bits that the algorithm
uses.
3 On each input behaviour of algorithm is random. Analyze
worst-case over all inputs of the (average) performance.

Fahad (IITH) CS5390 10 10 / 58

 Part II

Basics of Discrete Probability

Fahad (IITH) CS5390 11 11 / 58

Discrete Probability
We restrict attention to finite probability spaces.
Definition
A discrete probability space is a pair (⌦, Pr) consists of finite set ⌦
of elementary events and function p : ⌦ ! P[0, 1] which assigns a
probability Pr[!] for each ! 2 ⌦ such that !2⌦ Pr[!] = 1.

Fahad (IITH) CS5390 12 12 / 58

Discrete Probability
We restrict attention to finite probability spaces.
Definition
A discrete probability space is a pair (⌦, Pr) consists of finite set ⌦
of elementary events and function p : ⌦ ! P[0, 1] which assigns a
probability Pr[!] for each ! 2 ⌦ such that !2⌦ Pr[!] = 1.

Examples
An unbiased coin.

Fahad (IITH) CS5390 12 12 / 58

Discrete Probability
We restrict attention to finite probability spaces.
Definition
A discrete probability space is a pair (⌦, Pr) consists of finite set ⌦
of elementary events and function p : ⌦ ! P[0, 1] which assigns a
probability Pr[!] for each ! 2 ⌦ such that !2⌦ Pr[!] = 1.

Examples
An unbiased coin. ⌦ = {H, T } and Pr[H] = Pr[T ] = 1/2.

Fahad (IITH) CS5390 12 12 / 58

Discrete Probability
We restrict attention to finite probability spaces.
Definition
A discrete probability space is a pair (⌦, Pr) consists of finite set ⌦
of elementary events and function p : ⌦ ! P[0, 1] which assigns a
probability Pr[!] for each ! 2 ⌦ such that !2⌦ Pr[!] = 1.

Examples
An unbiased coin. ⌦ = {H, T } and Pr[H] = Pr[T ] = 1/2.
A 6-sided unbiased die.

Fahad (IITH) CS5390 12 12 / 58

Discrete Probability
We restrict attention to finite probability spaces.
Definition
A discrete probability space is a pair (⌦, Pr) consists of finite set ⌦
of elementary events and function p : ⌦ ! P[0, 1] which assigns a
probability Pr[!] for each ! 2 ⌦ such that !2⌦ Pr[!] = 1.

Examples
An unbiased coin. ⌦ = {H, T } and Pr[H] = Pr[T ] = 1/2.
A 6-sided unbiased die. ⌦ = {1, 2, 3, 4, 5, 6} and
Pr[i ] = 1/6 for all i 2 ⌦.

Fahad (IITH) CS5390 12 12 / 58

Discrete Probability
We restrict attention to finite probability spaces.
Definition
A discrete probability space is a pair (⌦, Pr) consists of finite set ⌦
of elementary events and function p : ⌦ ! P[0, 1] which assigns a
probability Pr[!] for each ! 2 ⌦ such that !2⌦ Pr[!] = 1.

Examples
An unbiased coin. ⌦ = {H, T } and Pr[H] = Pr[T ] = 1/2.
A 6-sided unbiased die. ⌦ = {1, 2, 3, 4, 5, 6} and
Pr[i ] = 1/6 for all i 2 ⌦.
A pair of independent dice.

Fahad (IITH) CS5390 12 12 / 58

Discrete Probability
We restrict attention to finite probability spaces.
Definition
A discrete probability space is a pair (⌦, Pr) consists of finite set ⌦
of elementary events and function p : ⌦ ! P[0, 1] which assigns a
probability Pr[!] for each ! 2 ⌦ such that !2⌦ Pr[!] = 1.

Examples
An unbiased coin. ⌦ = {H, T } and Pr[H] = Pr[T ] = 1/2.
A 6-sided unbiased die. ⌦ = {1, 2, 3, 4, 5, 6} and
Pr[i ] = 1/6 for all i 2 ⌦.
A pair of independent dice.
⌦ = {(i , j ) | 1  i  6, 1  j  6} and Pr[(i , j )] = 1/6
for all (i , j ) 2 ⌦.
Fahad (IITH) CS5390 12 12 / 58
Events
Definition
Given a probability space (⌦, Pr) an event is a subset of ⌦. In other
words an event is a collection of elementary
P events. The probability
of an event A, denoted by Pr[A], is !2A Pr[!].

The complement event of an event A ✓ ⌦ is the event ⌦ \ A

frequently denoted by Ā.

Fahad (IITH) CS5390 13 13 / 58

Events
Definition
Given a probability space (⌦, Pr) an event is a subset of ⌦. In other
words an event is a collection of elementary
P events. The probability
of an event A, denoted by Pr[A], is !2A Pr[!].

The complement event of an event A ✓ ⌦ is the event ⌦ \ A

frequently denoted by Ā.

Example
A pair of independent dice. ⌦ = {(i , j ) | 1  i  6, 1  j  6}.
Let A be the event that the sum of the two numbers on the dice is
even. n o
Then A = (i , j ) 2 ⌦ (i + j ) is even .
Pr[A] = |A|/36 = 1/2.
Fahad (IITH) CS5390 13 13 / 58
Independent Events
Definition
Given a probability space (⌦, Pr) and two events A, B are
independent if and only if Pr[A \ B] = Pr[A] Pr[B]. Otherwise
they are dependent. In other words A, B independent implies one
does not a↵ect the other.

Fahad (IITH) CS5390 14 14 / 58

Independent Events
Definition
Given a probability space (⌦, Pr) and two events A, B are
independent if and only if Pr[A \ B] = Pr[A] Pr[B]. Otherwise
they are dependent. In other words A, B independent implies one
does not a↵ect the other.

Example
Two coins. ⌦ = {HH, TT , HT , TH} and
Pr[HH] = Pr[TT ] = Pr[HT ] = Pr[TH] = 1/4.
1 A is the event that the first coin is heads and B is the event
that second coin is tails.

Fahad (IITH) CS5390 14 14 / 58

Independent Events
Definition
Given a probability space (⌦, Pr) and two events A, B are
independent if and only if Pr[A \ B] = Pr[A] Pr[B]. Otherwise
they are dependent. In other words A, B independent implies one
does not a↵ect the other.

Example
Two coins. ⌦ = {HH, TT , HT , TH} and
Pr[HH] = Pr[TT ] = Pr[HT ] = Pr[TH] = 1/4.
1 A is the event that the first coin is heads and B is the event
that second coin is tails. A, B are independent.

Fahad (IITH) CS5390 14 14 / 58

Independent Events
Definition
Given a probability space (⌦, Pr) and two events A, B are
independent if and only if Pr[A \ B] = Pr[A] Pr[B]. Otherwise
they are dependent. In other words A, B independent implies one
does not a↵ect the other.

Example
Two coins. ⌦ = {HH, TT , HT , TH} and
Pr[HH] = Pr[TT ] = Pr[HT ] = Pr[TH] = 1/4.
1 A is the event that the first coin is heads and B is the event
that second coin is tails. A, B are independent.
2 A is the event that both are not tails and B is event that second
coin is heads.

Fahad (IITH) CS5390 14 14 / 58

Independent Events
Definition
Given a probability space (⌦, Pr) and two events A, B are
independent if and only if Pr[A \ B] = Pr[A] Pr[B]. Otherwise
they are dependent. In other words A, B independent implies one
does not a↵ect the other.

Example
Two coins. ⌦ = {HH, TT , HT , TH} and
Pr[HH] = Pr[TT ] = Pr[HT ] = Pr[TH] = 1/4.
1 A is the event that the first coin is heads and B is the event
that second coin is tails. A, B are independent.
2 A is the event that both are not tails and B is event that second
coin is heads. A, B are dependent.

Fahad (IITH) CS5390 14 14 / 58

Union bound
The probability of the union of two events, is no bigger than the probability of the
sum of their probabilities.

Lemma
Forh any two
i eventsh Ei and F,h we
i have that
Pr E [ F  Pr E + Pr F .

Fahad (IITH) CS5390 15 15 / 58

Union bound
The probability of the union of two events, is no bigger than the probability of the
sum of their probabilities.

Lemma
Forh any two
i eventsh Ei and F,h we
i have that
Pr E [ F  Pr E + Pr F .

Proof.
h i X
Pr E [ F = Pr[x]
x2E[F
X X h i h i
 Pr[x] + Pr[x] = Pr E + Pr F .
x2E x2F

Fahad (IITH) CS5390 15 15 / 58

Law of Total Probability
Theorem
Let E1 , . .S
. , Er be mutually disjoint events in a a probability space
and ⌦ = i Ei . Then, for any event A,
r
X
Pr[A] = Pr[A \ Ei ] .
i =1

Fahad (IITH) CS5390 16 16 / 58

Example: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Randomized algorithm:
1 Pick a random n ⇥ 1 vector r 2 {0, 1}n .
2 Return the answer of the equality ABr = Cr .
3 Running time?

Fahad (IITH) CS5390 17 17 / 58

Example: Verifying Matrix Multiplication
Problem
Given three n ⇥ n matrices A, B, C is AB = C ?

Randomized algorithm:
1 Pick a random n ⇥ 1 vector r 2 {0, 1}n .
2 Return the answer of the equality ABr = Cr .
3 Running time? O(n2 )!
Theorem
If AB = C then the algorithm will always say YES. If AB 6= C then
the algorithm will say YES with probability at most 1/2. Can repeat
the algorithm 100 times independently to reduce the probability of a
false positive to 1/2100 .
Fahad (IITH) CS5390 17 17 / 58
Principle of deferred decisions
If AB = C then the algorithm will always say YES.

Fahad (IITH) CS5390 18 18 / 58

Principle of deferred decisions
If AB = C then the algorithm will always say YES.
Let D = AB C . If AB 6= C , then D 6= 0.

Fahad (IITH) CS5390 18 18 / 58

Principle of deferred decisions
If AB = C then the algorithm will always say YES.
Let D = AB C . If AB 6= C , then D 6= 0.
ABr = Cr implies that Dr = 0.

Fahad (IITH) CS5390 18 18 / 58

Principle of deferred decisions
If AB = C then the algorithm will always say YES.
Let D = AB C . If AB 6= C , then D 6= 0.
ABr = Cr implies that Dr = 0.
If AB 6= C , then we want to prove that Pr[Dr = 0]  12 .

Fahad (IITH) CS5390 18 18 / 58

Principle of deferred decisions
If AB = C then the algorithm will always say YES.
Let D = AB C . If AB 6= C , then D 6= 0.
ABr = Cr implies that Dr = 0.
If AB 6= C , then we want to prove that Pr[Dr = 0]  12 .
W.l.g the last entry in the first row of D is nonzero.

Fahad (IITH) CS5390 18 18 / 58

Principle of deferred decisions

Fahad (IITH) CS5390 19 19 / 58

Probability Amplification
Run the above algorithm 100 times and output YES, if all the
100 times we get YES. Otherwise output NO.

Fahad (IITH) CS5390 20 20 / 58

Probability Amplification
Run the above algorithm 100 times and output YES, if all the
100 times we get YES. Otherwise output NO.
If AB = C then the algorithm will always say YES.

Fahad (IITH) CS5390 20 20 / 58

Probability Amplification
Run the above algorithm 100 times and output YES, if all the
100 times we get YES. Otherwise output NO.
If AB = C then the algorithm will always say YES.
Otherwise, let a1 , . . . , a100 be the 100 random vectors in the
executions of the algorithm. Then, the failure probability

= Pr[ABa1 = Ca1 \ . . . \ ABa100 = Ca100 ]

= ⇧100
i =1 Pr[ABai = Cai ]
1
 100 .
2

Fahad (IITH) CS5390 20 20 / 58

Probability Amplification
Run the above algorithm 100 times and output YES, if all the
100 times we get YES. Otherwise output NO.
If AB = C then the algorithm will always say YES.
Otherwise, let a1 , . . . , a100 be the 100 random vectors in the
executions of the algorithm. Then, the failure probability

= Pr[ABa1 = Ca1 \ . . . \ ABa100 = Ca100 ]

= ⇧100
i =1 Pr[ABai = Cai ]
1
 100 .
2

1
Success probability is at least 1 2100
> 0.99

Fahad (IITH) CS5390 20 20 / 58