Chapter 2 - Asymptotic Notation (Notes)
Chapter 2 - Asymptotic Notation (Notes)
Chapter 2 - Asymptotic Notation (Notes)
Lecture Notes
Textbook: Introduction to the Design and Analysis of Algorithms 3rd Ed., Anany Levitin,
Pearson.
Section 1:
1. Introduction to Analysis of Algorithms Efficiency
Besides efficiency, generality and simplicity of algorithms are important to investigate, however,
efficiency can be studied in precise quantitative terms. Even for today’s computers, the efficiency
considerations are of primary importance from a practical point of view. So, efficiency of
algorithms is the consideration of this course.
o Some algorithms require more than one parameter to indicate the size of their
inputs (e.g., the number of vertices and the number of edges for algorithms on
graphs represented by their adjacency lists) size. In such situations, it is preferable
to measure size by the number b of bits in the n’s binary representation:
This metric usually gives a better idea about the efficiency of algorithms
in question.
➢ One possible approach is to count the number of times each of the algorithm’s
operations is executed. This approach is both excessively difficult and, as we shall see,
usually unnecessary. The thing to do is to identify the most important operation of the
algorithm, called the basic operation, the operation contributing the most to the total
running time, and compute the number of times the basic operation is executed.
➢ Thus, the established framework for the analysis of an algorithm’s time efficiency
suggests measuring it by counting the number of times the algorithm’s basic
operation is executed on inputs of size n.
➢ This formula should be used with caution. The count C(n) does not contain any
information about operations that are not basic, and, in fact, the count itself is often
computed only approximately. Further, the constant cop is also an approximation whose
reliability is not always easy to assess. Still, unless n is extremely large or very small, the
formula can give a reasonable estimate of the algorithm’s running time.
o It also makes it possible to answer such questions as “How much faster would this
algorithm run on a machine that is 10 times faster than the one we have?” The
answer is, obviously, 10 times.
1
o Assuming that 𝐶(𝑛) = 𝑛(𝑛 − 1), how much longer will the algorithm run if we
2
double its input size? The answer is about four times longer. Indeed, for all but
very small values of n,
Orders of Growth
➢ Why this emphasis on the count’s order of growth for large input sizes?
➢ A difference in running times on small inputs is not what really distinguishes efficient
algorithms from inefficient ones. When we have to compute, for example, the greatest
common divisor of two small numbers, it is not immediately clear how much more efficient
Euclid’s algorithm is compared to the other two algorithms discussed in the previous section
or even why we should care which of them is faster and by how much.
➢ For large values of n, it is the function’s order of growth that counts: just look at Table 2.1,
which contains values of a few functions particularly important for analysis of algorithms.
➢ The magnitude of the numbers in Table 2.1 has a profound significance for the analysis of
algorithms. The function growing the slowest among these is the logarithmic function. It
grows so slowly, in fact, that we should expect a program implementing an algorithm with a
logarithmic basic-operation count to run practically instantaneously on inputs of all realistic
sizes. Also note that although specific values of such a count depend, of course, on the
logarithm’s base, the formula
makes it possible to switch from one base to another, leaving the count logarithmic but with
a new multiplicative constant. This is why we omit a logarithm’s base and write simply log n
in situations where we are interested just in a function’s order of growth to within a
multiplicative constant.
➢ On the other end of the spectrum are the exponential function 2n and the factorial function n!
Both these functions grow so fast that their values become astronomically large even for
rather small values of n. (This is the reason why we did not include their values for n > 102
in Table 2.1.)
o For example, it would take about 4 x 1010 years for a computer making a trillion (1012)
operations per second to execute 2100 operations. Though this is incomparably faster
than it would have taken to execute 100! operations, it is still longer than 4.5 billion
(4.5 x 109) years—the estimated age of the planet Earth. There is a tremendous
difference between the orders of growth of the functions 2n and n!, yet both are often
referred to as “exponential-growth functions” (or simply “exponential”) despite the
fact that, strictly speaking, only the former should be referred to as such. The bottom
line, which is important to remember, is this:
➢ Another way to appreciate the qualitative difference among the orders of growth of the
functions in Table 2.1 is to consider how they react to, say, a twofold increase in the value
of their argument n. The function log2 n increases in value by just 1 (because log2 2n = log2
2 + log2 n = 1 + log2 n); the linear function increases twofold, the linearithmic function n
log2 n increases slightly more than twofold; the quadratic function n2 and cubic function n3
increase fourfold and eightfold, respectively (because (2n)2 = 4n2 and (2n)3 = 8n3); the
value of 2n gets squared (because 22n = (2n)2); and n! increases much more than that (yes,
even mathematics refuses to cooperate to give a neat answer for n!).
There are many algorithms for which running time depends not only on an input size but also on
the specifics of a particular input. Consider, as an example, sequential search. This is a
straightforward algorithm that searches for a given item (some search key K) in a list of n
elements by checking successive elements of the list until either a match with the search key is
found or the list is exhausted. Here is the algorithm’s pseudocode, in which, for simplicity, a list
is implemented as an array. It also assumes that the second condition A[i] = K will not be checked
if the first one, which checks that the array’s index does not exceed its upper bound, fails.
Clearly, the running time of this algorithm can be quite different for the same list size n. In the
worst case, when there are no matching elements or the first matching element happens to be
the last one on the list, the algorithm makes the largest number of key comparisons among all
possible inputs of size n: Cworst(n) = n.
Worst-Case
➢ The worst-case efficiency of an algorithm is its efficiency for the worst-case input of size
n, which is an input (or inputs) of size n for which the algorithm runs the longest among all
possible inputs of that size.
➢ The way to determine the worst-case efficiency of an algorithm is, in principle, quite
straightforward: analyse the algorithm to see what kind of inputs yield the largest value of
the basic operation’s count C(n) among all possible inputs of size n and then compute
this worst-case value Cworst(n).
➢ For sequential search, the answer was obvious.
➢ Clearly, the worst-case analysis provides very important information about an
algorithm’s efficiency by bounding its running time from above.
➢ In other words, it guarantees that for any instance of size n, the running time will not
exceed Cworst(n), its running time on the worst-case inputs.
Best-Case
➢ The best-case efficiency of an algorithm is its efficiency for the best-case input of size n,
which is an input (or inputs) of size n for which the algorithm runs the fastest among all
possible inputs of that size.
➢ Accordingly, we can analyse the best-case efficiency as follows.
➢ First, we determine the kind of inputs for which the count C(n) will be the smallest among
all possible inputs of size n.
(Note that the best-case does not mean the smallest input; it means the input of size n for
which the algorithm runs the fastest.)
➢ Then we ascertain the value of C(n) on these most convenient inputs.
For example, the best-case inputs for sequential search are lists of size n with their first
element equal to a search key; accordingly, Cbest(n) = 1 for this algorithm.
Average-Case
➢ It should be clear from our discussion, however, that neither the worst-case analysis nor its
best-case counterpart yields the necessary information about an algorithm’s behaviour on a
“typical” or “random” input. This is the information that the average-case efficiency seeks
to provide.
➢ To analyse the algorithm’s average-case efficiency, we must make some assumptions about
possible inputs of size n.
It should be clear from the preceding discussion that the average-case efficiency cannot be
obtained by taking the average of the worst-case and the best-case efficiencies. Even though this
average does occasionally coincide with the average-case cost, it is not a legitimate way of
performing the average-case analysis.
Does one really need the average-case efficiency information? The answer is unequivocally yes:
there are many important algorithms for which the average-case efficiency is much better than
the overly pessimistic worst-case efficiency would lead us to believe. So, without the average-
case analysis, computer scientists could have missed many important algorithms.
Yet another type of efficiency is called amortized efficiency. It applies not to a single run of an
algorithm but rather to a sequence of operations performed on the same data structure. It turns
out that in some situations a single operation can be expensive, but the total time for an entire
sequence of n such operations is always significantly better than the worst-case efficiency of that
single operation multiplied by n. So, we can “amortize” the high cost of such a worst-case
occurrence over the entire sequence in a manner similar to the way a business would amortize
the cost of an expensive item over the years of the item’s productive life.
Big-O, commonly written as O, is an Asymptotic Notation for the worst case, or the longest
amount of time an algorithm can possibly take to complete. It provides us with an asymptotic
upper bound for the growth rate of run-time of an algorithm.
Ο(f(n)) = { g(n) : there exists c > 0 and n0 such that f(n) ≤ c.g(n) for all
n > n0. }
2.2 Ω Notation (Big-Omega Notation)
Big-Omega, commonly written as Ω, is an Asymptotic Notation for the best case, or the best
amount of time an algorithm can possibly take to complete. It provides us with an asymptotic
lower bound for the growth rate of run-time of an algorithm.
Ω(f(n)) ≥ { g(n) : there exists c > 0 and n0 such that g(n) ≤ c.f(n) for all
n > n0. }
θ(f(n)) = { g(n) if and only if g(n) = Ο(f(n)) and g(n) = Ω(f(n)) for all n
> n0. }
3. Common Asymptotic Notations
Following is a list of some common asymptotic notations −
Notation Name
O(1) constant
O(log n) logarithmic
O(n) linear
O(n2) quadratic
O(n!) factorial
Here the best way to understand Big O thoroughly examples in code. So, below are some
common orders of growth along with descriptions and examples where possible.
1. O(1)
void printFirstElementOfArray(int arr[])
{
printf("First element of array = %d",arr[0]);
}
This function runs in O(1) time (or "constant time") relative to its input. The input array could
be 1 item or 1,000 items, but this function would still just require one step.
2. O(n)
void printAllElementOfArray(int arr[], int size)
{
for (int i = 0; i < size; i++)
{
printf("%d\n", arr[i]);
}
}
This function runs in O(n) time (or "linear time"), where n is the number of items in the array.
If the array has 10 items, we have to print 10 times. If it has 1000 items, we have to print 1000
times.
3. O(n2)
void printAllPossibleOrderedPairs(int arr[], int size)
{
for (int i = 0; i < size; i++)
{
for (int j = 0; j < size; j++)
{
printf("%d = %d\n", arr[i], arr[j]);
}
}
}
Here we're nesting two loops. If our array has n items, our outer loop runs n times and our inner
loop runs n times for each iteration of the outer loop, giving us n2 total prints. Thus this
function runs in O(n2) time (or "quadratic time"). If the array has 10 items, we have to print 100
times. If it has 1000 items, we have to print 1000000 times.
4. O(2n)
int fibonacci(int num)
{
if (num <= 1) return num;
return fibonacci(num - 2) + fibonacci(num - 1);
}
Why can we get away with this? Remember, for big O notation we're looking at what happens
as n gets arbitrarily large. As n gets really big, adding 100 or dividing by 2 has a decreasingly
significant effect.
6. Drop the less significant terms
void printAllNumbersThenAllPairSums(int arr[], int size)
{
for (int i = 0; i < size; i++)
{
printf("%d\n", arr[i]);
}
Similarly:
Again, we can get away with this because the less significant terms quickly become, well, less
significant as n gets big.
Here we might have 100 items in our array, but the first item might be the that element, in this
case we would return in just 1 iteration of our loop.
In general we'd say this is O(n) runtime and the "worst case" part would be implied. But to be
more specific we could say this is worst case O(n) and best case O(1) runtime. For some
algorithms we can also make rigorous statements about the "average case" runtime.
8. Other Examples
Let's take the following C example which contains a for loop, iterates from i = 0 to i <
10000 and prints each value of that i:
#include<stdio.h>
void print_values(int end)
{
for (int i = 0; i < end; i++)
{
printf("%d\n", i);
}
}
int main()
{
print_values(10000);
return 0;
}
We could put a timer at the beginning and the end of the line of code which calls this function,
this would then give us the running time of our print_values algorithm, right?
#include<stdio.h>
#include<time.h>
void print_values(int end)
{
for (int i = 0; i < end; i++)
{
printf("%d\n", i);
}
}
int main()
{
clock_t t;
t = clock();
print_values(10000);
return 0;
}
Maybe, but what if you run it again, three times, write down your results and then move to
another machine with a higher spec and run it another three times. I bet upon comparison of the
results you will get different running times!
This is where asymptotic notations are important. They provide us with a mathematical
foundation for representing the running time of our algorithms consistently.
We create this consistency by talking about operations our code has to perform. Operations
such as array lookups, print statements and variable assignments.
If we were to annotate print_values with the amount of times each line within the function is
executed for the input 10000, we would have something as follows:
void print_values(int end) //end = 10000
{
for (int i = 0; i < end; i++) //Execution count: 10000
{
printf("%d\n", i); // Execution count: 10000
}
}
If we were to change the input value of print_values function, our print statement would be
exercised more or less, depending on the value of that input.
If we were to put this into an arithmetic expression, we would get 10000+1, using intuition we
know that the 10000 is variable on the input size, if we call the input value n, we would now
have the expression n+1.
I could now argue that the worst case running time for print_values is O(n+1). n for the loop
block and 1 for the print statement.
In the grand scheme of things, the constant value 1 is pretty insignificant at the side of the
variable value n. So we simply reduce the above expression to O(n), and there we have our
Big-O running time of print_values.
As our code prints each and every value from 0 to the input , as the loop is the most significant
part of the code, we are able to say that our code is of running time O(n) where n is the variable
length of the array! Simples!
An algorithm of running time O(n) is said to be linear, which essentially means the algorithms
running time will increase linearly with its input (n).
9. Proving Big-O
We can prove, mathematically, that print_values is in-fact O(n), which brings us on to the
formal definition for Big-O:
f(n) = O(g(n)) if c and some initial value k are positive when f(n) <= c * g(n) for all n > k is
true.
We can turn this formal definition into an actual definition of our above code, which we can
then in turn prove.
c can be any integer while k is the amount of iterations we must perform for the expression to
be true for every subsequent value of n.
As c is just 1, we can simplify our expression to print_values <= n.
We can see that n must be greater than the value 0 of constant k in order to satisfy the
expression print_values <= n.
The above must be true for all values of n greater than k (0), so if n was 10, 10 <= 1 * 10 for 10
> 0 is also true.
What we're basically saying here is that no matter our input (n), it must be greater than or equal
to our constant (c) when the size of our input (n) is more than another constant value (k), in our
case the iteration count of the function.
But where do our constants come from? Well they are just values, we typically start at 1 and
work our way up to seek a constant which makes the expression f(n) <= c * g(n) for all n > k
true. If we cannot find such combination of constants, then our code does not have a running
time of O(n) and our hypothesis was incorrect.
10 Disproving Big-O
Lets take a new C function, which contains a for loop, iterates from i = 0 to i < 100 and an
another nested for loop from j = 0 to j < 100 which prints each value of that i and j:
Suppose our constant c is 1, 1 <= 1 * 1 for 1 > 0, this is true - however our definition says that
g(n) must be greater than all values of f(n).
So if we take the value 2 of n, 2 <= 1 * 4 for 1 > 0, we can see that this is now false, which
disproves our hypothesis that print_values_with_repeat is O(n). Even if we change our
constant c to 2, this would still prove false eventually.
Suppose our constant c is still 1, our expression would now be 3 <= 3 * 32 for 3 > 0, this is true,
great! print_values_with_repeat is in-fact O(n2).
O(n2) is a quadratic time algorithm, as the running time of the algorithm increases quadratically
to the input.
Asymptotic Notation 1
Big-O Notation
• Definition: f (n) = O(g(n)) iff there are two
positive constants c and n0 such that
|f (n)| ≤ c |g(n)| for all n ≥ n0
• If f (n) is nonnegative, we can simplify the last
condition to
0 ≤ f (n) ≤ c g(n) for all n ≥ n0
• We say that “f (n) is big-O of g(n).”
• As n increases, f (n) grows no faster than g(n).
In other words, g(n) is an asymptotic upper
bound on f (n).
cg(n)
f(n)
f(n) = O(g(n))
n0
Asymptotic Notation 3
Example: n2 + n = O(n3 )
Proof:
• Here, we have f (n) = n2 + n, and g(n) = n3
• Notice that if n ≥ 1, n ≤ n3 is clear.
• Also, notice that if n ≥ 1, n2 ≤ n3 is clear.
• Side Note: In general, if a ≤ b, then na ≤ nb
whenever n ≥ 1. This fact is used often in these
types of proofs.
• Therefore,
n2 + n ≤ n3 + n3 = 2n3
Big-Ω notation
• Definition: f (n) = Ω(g(n)) iff there are two
positive constants c and n0 such that
|f (n)| ≥ c |g(n)| for all n ≥ n0
• If f (n) is nonnegative, we can simplify the last
condition to
0 ≤ c g(n) ≤ f (n) for all n ≥ n0
• We say that “f (n) is omega of g(n).”
• As n increases, f (n) grows no slower than g(n).
In other words, g(n) is an asymptotic lower bound
on f (n).
f(n)
cg(n)
f(n) = O(g(n))
n0
Asymptotic Notation 5
Proof:
• Here, we have f (n) = n3 + 4n2 , and g(n) = n2
• It is not too hard to see that if n ≥ 0,
n3 ≤ n3 + 4n2
• Thus when n ≥ 1,
n2 ≤ n3 ≤ n3 + 4n2
• Therefore,
1n2 ≤ n3 + 4n2 for all n ≥ 1
Big-Θ notation
• Definition: f (n) = Θ(g(n)) iff there are three
positive constants c1 , c2 and n0 such that
c1 |g(n)| ≤ |f (n)| ≤ c2 |g(n)| for all n ≥ n0
• If f (n) is nonnegative, we can simplify the last
condition to
0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n) for all n ≥ n0
• We say that “f (n) is theta of g(n).”
• As n increases, f (n) grows at the same rate as
g(n). In other words, g(n) is an asymptotically
tight bound on f (n).
c2 g(n) f(n)
c1 g(n)
n0
Asymptotic Notation 7
Example: n2 + 5n + 7 = Θ(n2 )
Proof:
• When n ≥ 1,
n2 + 5n + 7 ≤ n2 + 5n2 + 7n2 ≤ 13n2
• When n ≥ 0,
n2 ≤ n2 + 5n + 7
• Thus, when n ≥ 1
1n2 ≤ n2 + 5n + 7 ≤ 13n2
Thus, we have shown that n2 + 5n + 7 = Θ(n2 )
(by definition of Big-Θ, with n0 = 1, c1 = 1, and
c2 = 13.)
Asymptotic Notation 8
Proof:
• Notice that if n ≥ 1,
1 2 1 7
n + 3n ≤ n2 + 3n2 = n2
2 2 2
• Thus,
1 2
n + 3n = O(n2 )
2
• Also, when n ≥ 0,
1 2 1 2
n ≤ n + 3n
2 2
• So
1 2
n + 3n = Ω(n2 )
2
• Since 21 n2 + 3n = O(n2 ) and 21 n2 + 3n = Ω(n2 ),
1 2
n + 3n = Θ(n2 )
2
Asymptotic Notation 12
Proof:
• We need to find positive constants c1 , c2 , and n0
such that
1 2
0 ≤ c1 n2 ≤ n − 3 n ≤ c2 n2 for all n ≥ n0
2
• Dividing by n2 , we get
1 3
0 ≤ c1 ≤ − ≤ c2
2 n
1 3
• c1 ≤ 2 − n holds for n ≥ 10 and c1 = 1/5
1 3
• 2 − n ≤ c2 holds for n ≥ 10 and c2 = 1.
• Thus, if c1 = 1/5, c2 = 1, and n0 = 10, then for
all n ≥ n0 ,
1 2
0 ≤ c1 n2 ≤ n − 3 n ≤ c2 n2 for all n ≥ n0 .
2
Thus we have shown that 12 n2 − 3n = Θ(n2 ).
Asymptotic Notation 14
log n n n log n n2 n3 2n
0 1 0 1 1 2
0.6931 2 1.39 4 8 4
1.099 3 3.30 9 27 8
1.386 4 5.55 16 64 16
1.609 5 8.05 25 125 32
1.792 6 10.75 36 216 64
1.946 7 13.62 49 343 128
2.079 8 16.64 64 512 256
2.197 9 19.78 81 729 512
2.303 10 23.03 100 1000 1024
2.398 11 26.38 121 1331 2048
2.485 12 29.82 144 1728 4096
2.565 13 33.34 169 2197 8192
2.639 14 36.95 196 2744 16384
2.708 15 40.62 225 3375 32768
2.773 16 44.36 256 4096 65536
2.833 17 48.16 289 4913 131072
2.890 18 52.03 324 5832 262144
log log m log m m
Asymptotic Notation 24
n 100n n2 11n2 n3 2n
1 100 1 11 1 2
2 200 4 44 8 4
3 300 9 99 27 8
4 400 16 176 64 16
5 500 25 275 125 32
6 600 36 396 216 64
7 700 49 539 343 128
8 800 64 704 512 256
9 900 81 891 729 512
10 1000 100 1100 1000 1024
11 1100 121 1331 1331 2048
12 1200 144 1584 1728 4096
13 1300 169 1859 2197 8192
14 1400 196 2156 2744 16384
15 1500 225 2475 3375 32768
16 1600 256 2816 4096 65536
17 1700 289 3179 4913 131072
18 1800 324 3564 5832 262144
19 1900 361 3971 6859 524288
Asymptotic Notation 25
n n2 n2 − n n2 + 99 n3 n3 + 234
2 4 2 103 8 242
6 36 30 135 216 450
10 100 90 199 1000 1234
14 196 182 295 2744 2978
18 324 306 423 5832 6066
22 484 462 583 10648 10882
26 676 650 775 17576 17810
30 900 870 999 27000 27234
34 1156 1122 1255 39304 39538
38 1444 1406 1543 54872 55106
42 1764 1722 1863 74088 74322
46 2116 2070 2215 97336 97570
50 2500 2450 2599 125000 125234
54 2916 2862 3015 157464 157698
58 3364 3306 3463 195112 195346
62 3844 3782 3943 238328 238562
66 4356 4290 4455 287496 287730
70 4900 4830 4999 343000 343234
74 5476 5402 5575 405224 405458
Asymptotic Notation
Polynomial Functions
40000
x
35000 x**2
x**3
30000 x**4
25000
20000
15000
10000
5000
0
0 5 10 15 20 25 30 35 40
26
Asymptotic Notation
Slow Growing Functions
250
log(x)
x
200 x*log(x)
x**2
150
100
50
0
0 5 10 15 20 25 30 35 40
27
Asymptotic Notation
Fast Growing Functions Part 1
5000
x
4500 x**3
4000 x**4
2**x
3500
3000
2500
2000
1500
1000
500
0
0 2 4 6 8 10
28
Asymptotic Notation
Fast Growing Functions Part 2
500000
x
450000 x**3
400000 x**4
2**x
350000
300000
250000
200000
150000
100000
50000
0
0 5 10 15 20
29
Asymptotic Notation
Why Constants and Non-Leading Terms Don’t Matter
4e+08
1000000*x
3.5e+08 300000*x**2 + 300*x
2**x
3e+08
2.5e+08
2e+08
1.5e+08
1e+08
5e+07
0
0 5 10 15 20 25 30
30
Asymptotic Notation 31
Classification Summary
2
√
c < log n < log n < n < n < n log n