CS161:

Design and Analysis of

Algorithms

Lecture 2
Leonidas Guibas 1
 Outline
Review of last lecture

Order of growth of functions

Asymptotic notations
Big O, big Ω, Θ, etc

Recurrences Slides modified from

• http://www.cs.virginia.edu/~luebke/cs332/

2
 Correctness of Algorithms

For any algorithm, we must prove that it

always returns the desired output for all
legal instances of the problem.
For sorting, this means even if (1) the
input is already sorted, or (2) it contains
repeated elements.
Algorithm correctness is NOT obvious in
many cases (e.g., optimization)

3
 Efficiency of Algorithms

Correctness alone is not sufficient

Brute-force algorithms exist for most
problems
To sort n numbers, we can enumerate all
permutations of these numbers and test
which permutation has the correct order
Why cannot we do this?
Too slow!
By what standard?
4
 Exact Algorithm Analysis is
Hard
Worst-case and average-case are difficult
to analyze precisely -- the details can be
very complicated

Easier to talk about upper and lower bounds on the

function T(n), the count of the number of operations
the algorithm performs. 5
 Kinds of Analyses

Worst case
Provides an upper bound on running time
An absolute guarantee
Best case – not very useful
Average case
Provides the expected running time
Very useful, but treat with care: what is “average”?
Random (equally likely) inputs
Real-life inputs

6
 Analysis of Insertion Sort

InsertionSort(A, n) {
for j = 2 to n {
key = A[j]
i = j - 1;
while (i > 0) and (A[i] > key) {
A[i+1] = A[i]
i = i - 1
}
How many times will
A[i+1] = key
this line execute?
}
}
7
 Analysis of Insertion Sort

InsertionSort(A, n) {
for j = 2 to n {
key = A[j]
i = j - 1;
while (i > 0) and (A[i] > key) {
A[i+1] = A[i]
i = i - 1
}
How many times will
A[i+1] = key
this line execute?
}
}
8
 Analysis of Insertion Sort
Statement cost time__
InsertionSort(A, n) {
for j = 2 to n { c1 n
key = A[j] c2 (n-1)
i = j - 1; c3 (n-1)
while (i > 0) and (A[i] > key) { c4 S
A[i+1] = A[i] c5 (S-(n-1))
i = i - 1 c6 (S-(n-1))
} 0
A[i+1] = key c7 (n-1)
} 0
}
S = t2 + t3 + … + tn where tj is number of while
expression evaluations for the jth for loop iteration 9
 Analyzing Insertion Sort
T(n) = c1n + c2(n-1) + c3(n-1) + c4S + c5(S - (n-1)) + c6(S - (n-1)) + c7(n-1)
= c8S + c9n + c10
What can S be?
Best case -- inner loop body never executed
tj = 1  S = n - 1
T(n) = an + b is a linear function
Worst case -- inner loop body executed for all previous
elements
tj = j  S = 2 + 3 + … + n = n(n+1)/2 - 1
T(n) = an2 + bn + c is a quadratic function
Average case
Can assume that on average, we have to insert A[j] into the
middle of A[1..j-1], so tj = j/2
S ≈ n(n+1)/4
T(n) is still a quadratic function
10
 Asymptotic Analysis
Abstract statement costs (don’t care about c1,
c2, etc)
Order of growth (as a function of n, the input
size) is the interesting measure:
Highest-order term is what counts
As the input size grows larger it is the high order term that
dominates x 10
4
4
100 * n
n2
3
T(n)

0 11
0 50 100 150 200
 Comparison of functions
log2n n nlog2n n2 n3 2n n!
10 3.3 10 33 102 103 103 106
102 6.6 102 660 104 106 1030 10158
103 10 103 104 106 109
104 13 104 105 108 1012
105 17 105 106 1010 1015
106 20 106 107 1012 1018

For a super computer that does 1 trillion operations

per second, it will be longer than 1 billion years
12
 Order of Growth

1 << log2n << n << nlog2n << n2 << n3 << 2n << n!

(We are slightly abusing of the “<<“ sign. It means

a smaller order of growth).

13
 Asymptotic Notations

We say InsertionSort’s worst-case

running time is Θ(n2)
Properly we should say running time is in
Θ(n2)
It is also in O(n2 )
What’s the relationships between Θ and O?
Formal definition comes next

14
 Asymptotic Notations

O: Big-Oh
Ω: Big-Omega
Θ: Theta
o: Small-oh
ω: Small-omega

1
 Big “O”
Informally, O(g(n)) is the set of all
functions with a smaller or same order of
growth as g(n), within a constant multiple
If we say f(n) is in O(g(n)), it means that
g(n) is an asymptotic upper bound on f(n)
Formally:
∃ C (>0) & n0, f(n) ≤ Cg(n) for ∀ n >= n0

What is O(n2)?
The set of all functions that grow slower than
or at the same order as n2
1
 Big “O”

So: O is an upper bound

n ∈ O(n2) notation, like ≤

n2 ∈ O(n2) We ignore constants,

lower order terms –
1000n ∈ O(n2) get to the essential
growth
n2 + n ∈ O(n2)
100n2 + n ∈ O(n2)
But: Even though formally we should
write n ∈ O(n2), in practice we write
1/1000 n3 ∉ O(n2) n = O(n2)

1
 Small “o”

Informally, o(g(n)) is the set of all functions with

a strictly smaller growth as g(n), within a
constant factor
What is o(n2)?
The set of all functions that grow slower than n2
So: o is a strict upper bound
notation, like <
1000n ∈ o(n2)
Formally,
But:
n2 ∉ o(n2) 𝑓𝑓 𝑛𝑛 ∈ 𝑜𝑜(𝑔𝑔 𝑛𝑛 )

𝑓𝑓 𝑛𝑛
→ 0 𝑎𝑎𝑎𝑎 𝑛𝑛 → ∞
𝑔𝑔 𝑛𝑛
1
 Big “Ω” [Omega]

Informally, Ω(g(n)) is the set of all functions with a larger

or same order of growth as g(n), within a constant
multiple
f(n) ∈ Ω(g(n)) means g(n) is an asymptotic lower bound
of f(n)
Intuitively, it is like f(n) ≥ g(n) Intuitively, Ω is like ≥

a lower bound notation

So:
n2 ∈ Ω(n)
1/1000 n2 ∈ Ω(n)
But:
1000 n ∉ Ω(n2)
1
 Small “ω” [omega]

Informally, ω(g(n)) is the set of all

functions with a strictly larger order of
growth than g(n), within a constant factor
So: Intuitively, ω is like >
n2 ∈ ω(n)
a strict lower bound
1/1000 n2 ∈ ω(n)
n2 ∉ ω(n2)

2
 Theta (“Θ”): Θ = O and Ω

Informally, Θ(g(n)) is the set of all

functions with the same order of growth
as g(n), within a constant multiple
f(n) ∈ Θ(g(n)) means g(n) is an
asymptotically tight bound on f(n)
Intuitively, it is like f(n) = g(n)
What is Θ(n2)?
The set of all functions that grow at the same
order as n2
21
 Big “Θ” [Theta]

So:
n2 ∈ Θ(n2)
n2 + n ∈ Θ(n2) Intuitively, Θ is like =
100n2 + n ∈ Θ(n2)
100n2 + log2n ∈ Θ(n2)
But:
nlog2n ∉ Θ(n2)
1000n ∉ Θ(n2)
1/1000 n3 ∉ Θ(n2)
22
 Tricky Cases

How about sqrt(n) and log2 n?

How about log2 n and log10 n

How about 2n and 3n

How about 3n and n!?

23
 Big “O”, Formally
There exist
Definition: For all

O(g(n)) = {f(n): ∃ positive constants C and n0

such that 0 ≤ f(n) ≤ Cg(n) ∀ n>n0}

lim n→∞ g(n)/f(n) > 0 (if the limit exists)

Abuse of notation (for convenience):

f(n) = O(g(n)) actually means f(n) ∈ O(g(n))

24
 Big “O”, Example

Claim: f(n) = 3n2 + 10n + 5 ∈ O(n2)

Proof from the definition
To prove this claim by definition, we need to find some positive
constants C and n0 such that f(n) <= Cn2 for all n > n0.
(Note: you just need to find one concrete example of c and n0
satisfying the condition.)
3n2 + 10n + 5 ≤ 10n2 + 10n + 10
≤ 10n2 + 10n2 + 10n2,∀ n ≥ 1
≤ 30 n2, ∀ n ≥ 1
Therefore, if we let C = 30 and n0 = 1, we have f(n) ≤ C n2, ∀ n ≥
n0.
Hence according to the definition of big-Oh, f(n) = O(n2).
Alternatively, we can show that
lim n→∞ n2 / (3n2 + 10n + 5) = 1/3 > 0

25
 Big “Ω”, Formally

Definition:
Ω(g(n)) = {f(n): ∃ positive constants C and n0
such that 0 ≤ Cg(n) ≤ f(n) ∀ n>n0}
lim n→∞ f(n)/g(n) > 0 (if the limit exists.)
Abuse of notation (for convenience):
f(n) = Ω(g(n)) actually means f(n) ∈ Ω(g(n))

26
 Big “Ω”, Example
Claim: f(n) = n2 / 10 = Ω(n)

Proof from the definition:

f(n) = n2 / 10, g(n) = n
Need to find a C and a n0 to satisfy the definition of f(n) ∈
Ω(g(n)), i.e., f(n) ≥ Cg(n) for n > n0
n ≤ n2 / 10 when n ≥ 10
If we let C = 1 and n0 = 10, we have f(n) ≥ Cn, ∀ n ≥ n0.
Therefore, according to the definition, f(n) = Ω(n).

Alternatively:
lim n→∞ f(n)/g(n) = lim n→∞ (n/10) = ∞

27
 Big “Θ”, Formally

Definition:
Θ(g(n)) = {f(n): ∃ positive constants c1, c2,
and n0 such that 0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n),
∀ n ≥ n0 }
lim n→∞ f(n)/g(n) = c > 0 and c < ∞
f(n) = O(g(n)) and f(n) = Ω(g(n))
Abuse of notation (for convenience):
f(n) = Θ(g(n)) actually means f(n) ∈ Θ(g(n))
Θ(1) means constant time.
28
 Big “Θ”, Example

Claim: f(n) = 2n2 + n = Θ (n2)

Proof from the definition:
Need to find the three constants c1, c2, and
n0 such that
c1n2 ≤ 2n2+n ≤ c2n2 for all n > n0
A simple solution is c1 = 2, c2 = 3, and n0 = 1

Alternatively, limn→∞(2n2+n)/n2 = 2
29
 More Examples
Prove n2 + 3n + lg n is in O(n2)
Want to find c and n0 such that
n2 + 3n + lg n <= cn2 for n > n0
Proof:
n2 + 3n + lg n <= 3n2 + 3n + 3lgn for n > 1
<= 3n2 + 3n2 + 3n2
<= 9n2
Or n2 + 3n + lg n <= n2 + n2 + n2 for n > 10
<= 3n2
30
 More Examples
Prove n2 + 3n + lg n is in Ω(n2)
Want to find c and n0 such that
n2 + 3n + lg n >= cn2 for n > n0

n2 + 3n + lg n >= n2 for n > 1

n2 + 3n + lg n = O(n2) and n2 + 3n + lg n = Ω (n2)

=> n2 + 3n + lg n = Θ(n2)

31
 O, Ω, and Θ

The definitions imply a constant n0 beyond which they are

satisfied. We do not care about small values of n.

32
Using Limits to Compare Orders of
Growth

f(n) ∈ o(g(n))
0 f(n) ∈ O(g(n))
lim f(n) / g(n) = c >0 f(n) ∈ Θ (g(n))
n→∞
∞ f(n) ∈ Ω(g(n))
f(n) ∈ ω (g(n))

33
 Logarithms

compare log2n and log10n

logab = logcb / logca

log2n = log10n / log102 ~ 3.3 log10n
Therefore lim(log2n / log10 n) = 3.3
log2n = Θ (log10n)

34
 Exponentials

Compare 2n and 3n
lim 2n / 3n = lim(2/3)n = 0
n→∞ n→∞
Therefore, 2n ∈ o(3n), and 3n ∈ ω(2n)

How about 2n and 2n+1?

2n / 2n+1 = ½, therefore 2n = Θ (2n+1)

35
 L’ Hopital’s Rule

Condition:
lim f(n) / g(n) = lim f’(n) / g’(n)
n→∞ n→∞ If both lim f(n) and
lim g(n) are ∞ or 0

You can apply this transformation as many

times as you want, as long as the condition
holds
36
Compare n0.5 and log n

lim n0.5 / ln n = ?
n→∞

(n0.5)’ = 0.5 1/n0.5

(ln n)’ = 1 / n
lim (1/n0.5 / 1/n) = lim (n0.5) = ∞
Therefore, ln n ∈ o(n0.5)
In fact, ln n ∈ o(nε), for any ε > 0 – and so is log n

37
Stirling’s Formula (Useful)
n
n
n! ≈ 2π n   = 2π n n +1 / 2 − n
e
e
n +1 / 2 − n
n! ≈ (constant) n e

38
 n
n! c nnn  n 
lim n = lim n n = lim c n   = ∞
Compare 2n and n! n →∞ 2 n →∞ 2 e n →∞
 2e 

Therefore, 2n = o(n!)
n! c nn n c n
lim n = lim n n = lim n = 0
Compare nn and n! n →∞ n n →∞ n e n →∞ e

Therefore, nn = ω(n!)

How about log (n!)?

39
 n
c nn n +1 / 2
log(n! ) = log n
= C + log n − log(e )
n

e
1
= C + n log n + log n − n
2
n n 1
= C + log n + ( log n − n ) + log n
2 2 2
= Θ( n log n )
40
More Advanced Dominance
Rankings

41
Asymptotic Notation Summary

O: Big-Oh
Ω: Big-Omega
Θ: Theta
o: Small-oh
ω: Small-omega
Intuitively:
O is like ≤ Ω is like ≥ Θ is like =
o is like < ω is like >
42
Properties of Asymptotic Notations

CLRS textbook, page 51

Transitivity
f(n) = Θ(g(n)) and g(n) = Θ(h(n))
=> f(n) = Θ(h(n))
(holds true for o, O, ω, and Ω as well).
Symmetry
f(n) = Θ(g(n)) if and only if g(n) = Θ(f(n))
Transpose symmetry
f(n) = O(g(n)) if and only if g(n) = Ω(f(n))
f(n) = o(g(n)) if and only if g(n) = ω(f(n))
43
 Exponential and Logarithmic
Functions
CLRS textbook, pages 55-56
It is important to understand what logarithms are and
where they come from.
A logarithm is simply an inverse exponential function.
Saying bx = y is equivalent to saying that
x = logb y.
Logarithms reflect how many times we can double
something until we get to n, or halve something until we
get to 1.
log21 = ?
log22 = ?
44
 Useful Rules for Logarithms
For all a > 0, b > 0, c > 0, the following rules
hold
logba = logca / logcb = lg a / lg b
logban = n logba
log a
b b = a
log (ab) = log a + log b
lg (2n) = ?
log (a/b) = log (a) – log(b)
lg (n/2) = ?
lg (1/n) = ?
logba = 1 / logab
45
Useful Rules for Exponentials

For all a > 0, b > 0, c > 0, the following

rules hold
a0 = 1 (00 = ?)
a1 = a
a-1 = 1/a
(am)n = amn
(am)n = (an)m
aman = am+n
46
47
Analyzing Recursive Algorithms

48
 Recursive Algorithms
General idea:
Divide a large problem into smaller ones
By a constant ratio
By a constant or some variable
Solve each smaller one recursively or
explicitly
Combine the solutions of smaller ones to
form a solution for the original problem

Divide and Conquer

49
 MergeSort

MERGE-SORT A[1 . . n]
1. If n = 1, done.
2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .
3. “Merge” the 2 sorted lists.

Key subroutine: MERGE

50
Merging Two Sorted Arrays
Subarray 1 Subarray 2

20 12
13 11
7 9
2 1

51
Merging Two Sorted Arrays
Subarray 1 Subarray 2

20 12
13 11
7 9
2 1

52
 Merging Two Sorted Arrays
20 12
13 11
7 9
2 1

53
 Merging Two Sorted Arrays
20 12
13 11
7 9
2 1

54
 Merging Two Sorted Arrays
20 12
13 11
7 9
2 1

55
 Merging Two Sorted Arrays
20 12 20 12
13 11 13 11
7 9 7 9
2 1 2

56
 Merging Two Sorted Arrays
20 12 20 12
13 11 13 11
7 9 7 9
2 1 2

1 2

57
 Merging Two Sorted Arrays
20 12 20 12 20 12
13 11 13 11 13 11
7 9 7 9 7 9
2 1 2

1 2

58
 Merging Two Sorted Arrays
20 12 20 12 20 12
13 11 13 11 13 11
7 9 7 9 7 9
2 1 2

1 2 7

59
 Merging Two Sorted Arrays
20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2

1 2 7

60
 Merging Two Sorted Arrays
20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2

1 2 7 9

61
 Merging Two Sorted Arrays
20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2

1 2 7 9

62
 Merging Two Sorted Arrays
20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2

1 2 7 9 11

63
 Merging Two Sorted Arrays
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2

1 2 7 9 11

64
 Merging Two Sorted Arrays
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2

1 2 7 9 11 12

65
How to Show the Correctness of a
Recursive Algorithm?

By induction:
Base case: prove it works for small examples
Inductive hypothesis: assume the solution is
correct for all sub-problems
Step: show that, if the inductive hypothesis is
correct, then the algorithm is correct for the
original problem.

66
 Correctness of MergeSort
MERGE-SORT A[1 . . n]
1. If n = 1, done.
2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .
3. “Merge” the 2 sorted lists.
Proof:
1. Base case: if n = 1, the algorithm will return the correct answer
because A[1..1] is already sorted.
2. Inductive hypothesis: assume that the algorithm correctly sorts
A[1.. n/2 ] and A[n/2+1..n].
3. Step: if A[1.. n/2 ] and A[n/2+1..n] are both correctly sorted, the
whole array A[1.. n/2 ] and A[n/2+1..n] is sorted after merging.

67
How to Analyze the Time-Efficiency
of a Recursive Algorithm?

Express the running time on input of size

n as a function of the running time on
smaller problems

68
 Analyzing MergeSort

T(n) MERGE-SORT A[1 . . n]

Θ(1) 1. If n = 1, done.
2T(n/2) 2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .
f(n) 3. “Merge” the 2 sorted lists
Sloppiness: Should be T( n/2 ) + T( n/2 ) ,
but it turns out not to matter asymptotically.

69
 Analyzing MergeSort
1. Divide: Trivial.
2. Conquer: Recursively sort 2 subarrays.
3. Combine: Merge two sorted subarrays
T(n) = 2 T(n/2) + f(n) +Θ(1)
# subproblems Dividing and
subproblem size
Combining
1. What is the time for the base case? Constant
2. What is f(n)?
3. What is the growth order of T(n)?
70
 Merging Two Sorted Arrays
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2

1 2 7 9 11 12

Θ(n) time to merge a total of n elements (linear time).

71
 Recurrence for MergeSort
Θ(1) if n = 1;
T(n) =
2T(n/2) + Θ(n) if n > 1.
• Later we shall often omit stating the base
case when T(n) = Θ(1) for sufficiently
small n, but only when it has no effect on
the asymptotic solution to the recurrence.

• But what does T(n) solve to? i.e., is it O(n)

or O(n2) or O(n3) or …?

72
 Binary Search
To find an element in a sorted array, we
1. Check the middle element
2. If ==, we’ve found it
3. Else, if less than wanted, search right
half
4. else search left half
Example: Find 9
3 5 7 8 9 12 15
73
 Binary Search
To find an element in a sorted array, we
1. Check the middle element
2. If ==, we’ve found it
3. Else, if less than wanted, search right
half
4. else search left half
Example: Find 9
3 5 7 8 9 12 15
74
 Binary Search
To find an element in a sorted array, we
1. Check the middle element
2. If ==, we’ve found it
3. Else, if less than wanted, search right
half
4. else search left half
Example: Find 9
3 5 7 8 9 12 15
75
 Binary Search
To find an element in a sorted array, we
1. Check the middle element
2. If ==, we’ve found it
3. Else, if less than wanted, search right
half
4. else search left half
Example: Find 9
3 5 7 8 9 12 15
76
 Binary Search
To find an element in a sorted array, we
1. Check the middle element
2. If ==, we’ve found it
3. Else, if less than wanted, search right
half
4. else search left half
Example: Find 9
3 5 7 8 9 12 15
77
 Binary Search
To find an element in a sorted array, we
1. Check the middle element
2. If ==, we’ve found it
3. Else, if less than wanted, search right
half
4. else search left half
Example: Find 9
3 5 7 8 9 12 15
78
 Binary Search

BinarySearch (A[1..N], value) {

if (N == 0)
return -1; // not found
mid = (1+N)/2;
if (A[mid] == value)
return mid; // found
else if (A[mid] < value)
return BinarySearch (A[mid+1, N], value)
else
return BinarySearch (A[1..mid-1], value);
}
What’s the recurrence relation for its running time?
79
Recurrence for Binary Search

n
T ( n ) = T   + Θ(1)
2

T (1) = Θ(1)

80
 Recursive InsertionSort

RecursiveInsertionSort(A[1..n])
1. if (n == 1) do nothing;
2. RecursiveInsertionSort(A[1..n-1]);
3. Find index i in A such that A[i] <= A[n] < A[i+1];
4. Insert A[n] after A[i];

81
Recurrence for InsertionSort

T ( n ) = T (n − 1) + Θ( n )

T (1) = Θ(1)

82
 Compute Factorial

Factorial (n)
if (n == 1) return 1;
return n * Factorial (n-1);

• Note: here we use n as the size of the input. However,

usually for such algorithms we would use log(n), i.e., the
bits needed to represent n, as the input size.

83
 Recurrence for Computing
Factorial

T ( n ) = T (n − 1) + Θ(1)
T (1) = Θ(1)

Note: here we use n as the size of the input. However,

usually for such algorithms we would use log(n), i.e., the
bits needed to represent n, as the input size.

84
 What do These Signify?
T (n ) = T (n − 1) + 1

T (n ) = T (n − 1) + n

T (n ) = T (n / 2 ) + 1

T (n ) = 2T (n / 2 ) + 1

Challenge: how to solve the recurrence to get a closed

form, e.g. T(n) = Θ (n2) or T(n) = Θ(nlgn), or at least some
bound such as T(n) = O(n2)?

85
 Solving Recurrences
Running time of many algorithms can be
expressed in one of the following two
recursive forms
T ( n ) = aT ( n − b) + f ( n )
or

T ( n ) = aT (n / b) + f (n )

Both can be very hard to solve. We focus on

relatively easy ones, which you will encounter
frequently in many real algorithms (and exams…)
86
 Solving Recurrences

1. Recursion tree / iteration method

2. Substitution method
3. Master method

87