CMP 3005 – Analysis of Algorithms

Lecture Notes

Textbook: Introduction to the Design and Analysis of Algorithms 3rd Ed., Anany Levitin,
Pearson.

Section 1:
1. Introduction to Analysis of Algorithms Efficiency

➢ What does Analysis of Algorithms mean?

The term “analysis of algorithms” is usually used in a technical sense to mean an investigation
of an algorithm’s efficiency with respect to two resources: running time and memory space.

Besides efficiency, generality and simplicity of algorithms are important to investigate, however,
efficiency can be studied in precise quantitative terms. Even for today’s computers, the efficiency
considerations are of primary importance from a practical point of view. So, efficiency of
algorithms is the consideration of this course.

1. The Analysis Framework

2. Asymptotic Notations and Basic Efficiency Classes
3. Mathematical Analysis of Non-recursive and Recursive Algorithms
4. Empirical Analysis of Algorithms
1. The Analysis Framework
➢ There are two kind of efficiency.
o Time efficiency or Time complexity
o Space efficiency or Space complexity

➢ Time efficiency indicates how fast an algorithm in question runs.

➢ Space efficiency complexity, refers to the amount of memory units required by the
algorithm in addition to the space needed for its input and output.

➢ Main concern is time efficiency but space efficiency is important as well.

Measuring an Input’s Size

➢ For having a correct analysis framework, first, the effect of input size on the time
complexity of an algorithm should be understood.
➢ It is obvious that almost all algorithms run longer on larger inputs.
o For example, it takes longer to sort larger arrays, multiply larger matrices, and so
on.
➢ Therefore, it is logical to investigate an algorithm’s efficiency as a function of some
parameter n indicating the algorithm’s input size. In most cases, selecting such a
parameter is quite straightforward.
o For example, it will be the size of the list for problems of sorting, searching,
finding the list’s smallest element, and most other problems dealing with lists.
➢ For the problem of evaluating a polynomial p(x) = an xn + . . . + a0 of degree n, it will be
the polynomial’s degree or the number of its coefficients, which is larger by 1 than its
degree. You’ll see from the discussion that such a minor difference is inconsequential for
the efficiency analysis.
➢ The choice of an appropriate size metric can be influenced by operations of the algorithm
in question.
o For example, how should we measure an input’s size for a spell-checking
algorithm?
o If the algorithm examines individual characters of its input, we should measure
the size by the number of characters;
o if it works by processing words, we should count their number in the input.
 o We should make a special note about measuring input size for algorithms solving
problems such as checking primality of a positive integer n. Here, the input is just
one number, and it is this number’s magnitude that determines the input 1.

o Some algorithms require more than one parameter to indicate the size of their
inputs (e.g., the number of vertices and the number of edges for algorithms on
graphs represented by their adjacency lists) size. In such situations, it is preferable
to measure size by the number b of bits in the n’s binary representation:

This metric usually gives a better idea about the efficiency of algorithms
in question.

Units for Measuring Running Time

➢ The next issue concerns units for measuring an algorithm’s running time. Of course, we
can simply use some standard unit of time measurement—a second, or millisecond, and
so on—to measure the running time of a program implementing the algorithm. There are
obvious drawbacks to such an approach, however: dependence on the speed of a
particular computer, dependence on the quality of a program implementing the algorithm
and of the compiler used in generating the machine code, and the difficulty of clocking
the actual running time of the program. Since we are after a measure of an algorithm’s
efficiency, we would like to have a metric that does not depend on these extraneous
factors.

➢ One possible approach is to count the number of times each of the algorithm’s
operations is executed. This approach is both excessively difficult and, as we shall see,
usually unnecessary. The thing to do is to identify the most important operation of the
algorithm, called the basic operation, the operation contributing the most to the total
running time, and compute the number of times the basic operation is executed.

➢ As a rule, it is not difficult to identify the basic operation of an algorithm: it is usually

the most time-consuming operation in the algorithm’s innermost loop.
 o For example, most sorting algorithms work by comparing elements (keys) of a
list being sorted with each other; for such algorithms, the basic operation is a key
comparison.
o As another example, algorithms for mathematical problems typically involve
some or all of the four arithmetical operations: addition, subtraction,
multiplication, and division. Of the four, the most time-consuming operation is
division, followed by multiplication and then addition and subtraction, with
the last two usually considered together.

➢ Thus, the established framework for the analysis of an algorithm’s time efficiency
suggests measuring it by counting the number of times the algorithm’s basic
operation is executed on inputs of size n.

➢ Here is an important application.

o Let cop be the execution time of an algorithm’s basic operation on a particular
computer, and
o Let C(n) be the number of times this operation needs to be executed for this
algorithm.
o Then we can estimate the running time T(n) of a program implementing this
algorithm on that computer by the formula.

➢ This formula should be used with caution. The count C(n) does not contain any
information about operations that are not basic, and, in fact, the count itself is often
computed only approximately. Further, the constant cop is also an approximation whose
reliability is not always easy to assess. Still, unless n is extremely large or very small, the
formula can give a reasonable estimate of the algorithm’s running time.
o It also makes it possible to answer such questions as “How much faster would this
algorithm run on a machine that is 10 times faster than the one we have?” The
answer is, obviously, 10 times.
1
o Assuming that 𝐶(𝑛) = 𝑛(𝑛 − 1), how much longer will the algorithm run if we
2

double its input size? The answer is about four times longer. Indeed, for all but
very small values of n,
Orders of Growth
➢ Why this emphasis on the count’s order of growth for large input sizes?
➢ A difference in running times on small inputs is not what really distinguishes efficient
algorithms from inefficient ones. When we have to compute, for example, the greatest
common divisor of two small numbers, it is not immediately clear how much more efficient
Euclid’s algorithm is compared to the other two algorithms discussed in the previous section
or even why we should care which of them is faster and by how much.

➢ For large values of n, it is the function’s order of growth that counts: just look at Table 2.1,
which contains values of a few functions particularly important for analysis of algorithms.
➢ The magnitude of the numbers in Table 2.1 has a profound significance for the analysis of
algorithms. The function growing the slowest among these is the logarithmic function. It
grows so slowly, in fact, that we should expect a program implementing an algorithm with a
logarithmic basic-operation count to run practically instantaneously on inputs of all realistic
sizes. Also note that although specific values of such a count depend, of course, on the
logarithm’s base, the formula

makes it possible to switch from one base to another, leaving the count logarithmic but with
a new multiplicative constant. This is why we omit a logarithm’s base and write simply log n
in situations where we are interested just in a function’s order of growth to within a
multiplicative constant.

➢ On the other end of the spectrum are the exponential function 2n and the factorial function n!
Both these functions grow so fast that their values become astronomically large even for
rather small values of n. (This is the reason why we did not include their values for n > 102
in Table 2.1.)
o For example, it would take about 4 x 1010 years for a computer making a trillion (1012)
operations per second to execute 2100 operations. Though this is incomparably faster
than it would have taken to execute 100! operations, it is still longer than 4.5 billion
(4.5 x 109) years—the estimated age of the planet Earth. There is a tremendous
difference between the orders of growth of the functions 2n and n!, yet both are often
referred to as “exponential-growth functions” (or simply “exponential”) despite the
fact that, strictly speaking, only the former should be referred to as such. The bottom
line, which is important to remember, is this:

Algorithms that require an exponential number of operations are practical

for solving only problems of very small sizes.

➢ Another way to appreciate the qualitative difference among the orders of growth of the
functions in Table 2.1 is to consider how they react to, say, a twofold increase in the value
of their argument n. The function log2 n increases in value by just 1 (because log2 2n = log2
2 + log2 n = 1 + log2 n); the linear function increases twofold, the linearithmic function n
 log2 n increases slightly more than twofold; the quadratic function n2 and cubic function n3
increase fourfold and eightfold, respectively (because (2n)2 = 4n2 and (2n)3 = 8n3); the
value of 2n gets squared (because 22n = (2n)2); and n! increases much more than that (yes,
even mathematics refuses to cooperate to give a neat answer for n!).

Worst-Case, Best-Case, and Average-Case Efficiencies

There are many algorithms for which running time depends not only on an input size but also on
the specifics of a particular input. Consider, as an example, sequential search. This is a
straightforward algorithm that searches for a given item (some search key K) in a list of n
elements by checking successive elements of the list until either a match with the search key is
found or the list is exhausted. Here is the algorithm’s pseudocode, in which, for simplicity, a list
is implemented as an array. It also assumes that the second condition A[i] = K will not be checked
if the first one, which checks that the array’s index does not exceed its upper bound, fails.

Clearly, the running time of this algorithm can be quite different for the same list size n. In the
worst case, when there are no matching elements or the first matching element happens to be
the last one on the list, the algorithm makes the largest number of key comparisons among all
possible inputs of size n: Cworst(n) = n.
Worst-Case
➢ The worst-case efficiency of an algorithm is its efficiency for the worst-case input of size
n, which is an input (or inputs) of size n for which the algorithm runs the longest among all
possible inputs of that size.
➢ The way to determine the worst-case efficiency of an algorithm is, in principle, quite
straightforward: analyse the algorithm to see what kind of inputs yield the largest value of
the basic operation’s count C(n) among all possible inputs of size n and then compute
this worst-case value Cworst(n).
➢ For sequential search, the answer was obvious.
➢ Clearly, the worst-case analysis provides very important information about an
algorithm’s efficiency by bounding its running time from above.
➢ In other words, it guarantees that for any instance of size n, the running time will not
exceed Cworst(n), its running time on the worst-case inputs.

Best-Case
➢ The best-case efficiency of an algorithm is its efficiency for the best-case input of size n,
which is an input (or inputs) of size n for which the algorithm runs the fastest among all
possible inputs of that size.
➢ Accordingly, we can analyse the best-case efficiency as follows.
➢ First, we determine the kind of inputs for which the count C(n) will be the smallest among
all possible inputs of size n.
(Note that the best-case does not mean the smallest input; it means the input of size n for
which the algorithm runs the fastest.)
➢ Then we ascertain the value of C(n) on these most convenient inputs.
For example, the best-case inputs for sequential search are lists of size n with their first
element equal to a search key; accordingly, Cbest(n) = 1 for this algorithm.

Average-Case
➢ It should be clear from our discussion, however, that neither the worst-case analysis nor its
best-case counterpart yields the necessary information about an algorithm’s behaviour on a
“typical” or “random” input. This is the information that the average-case efficiency seeks
to provide.
➢ To analyse the algorithm’s average-case efficiency, we must make some assumptions about
possible inputs of size n.
It should be clear from the preceding discussion that the average-case efficiency cannot be
obtained by taking the average of the worst-case and the best-case efficiencies. Even though this
average does occasionally coincide with the average-case cost, it is not a legitimate way of
performing the average-case analysis.

Does one really need the average-case efficiency information? The answer is unequivocally yes:
there are many important algorithms for which the average-case efficiency is much better than
the overly pessimistic worst-case efficiency would lead us to believe. So, without the average-
case analysis, computer scientists could have missed many important algorithms.
Yet another type of efficiency is called amortized efficiency. It applies not to a single run of an
algorithm but rather to a sequence of operations performed on the same data structure. It turns
out that in some situations a single operation can be expensive, but the total time for an entire
sequence of n such operations is always significantly better than the worst-case efficiency of that
single operation multiplied by n. So, we can “amortize” the high cost of such a worst-case
occurrence over the entire sequence in a manner similar to the way a business would amortize
the cost of an expensive item over the years of the item’s productive life.

1. The Analysis Framework

2. Asymptotic Notations and Basic Efficiency Classes
3. Mathematical Analysis of Non-recursive and Recursive Algorithms
4. Empirical Analysis of Algorithms
Asymptotic Notations

1. What is Asymptotic Notations?

Asymptotic Notations are languages that allow us to analyse an algorithm’s run-time
performance. Asymptotic Notations identify running time by algorithm behaviour as the input
size for the algorithm increases. This is also known as an algorithm’s growth rate. Usually, the
time required by an algorithm falls under three types −

• Best Case − Minimum time required for program execution

• Average Case − Average time required for program execution.
• Worst Case − Maximum time required for program execution.

2. Types of Asymptotic Notation

Following are the commonly used asymptotic notations to calculate the running time
complexity of an algorithm.

• Ο Notation (Big-O Notation)

• Ω Notation (Big-Omega Notation)
• θ Notation (Theta Notation)

2.1 Ο Notation (Big-O Notation)

Big-O, commonly written as O, is an Asymptotic Notation for the worst case, or the longest
amount of time an algorithm can possibly take to complete. It provides us with an asymptotic
upper bound for the growth rate of run-time of an algorithm.

For example, for a function f(n)

Ο(f(n)) = { g(n) : there exists c > 0 and n0 such that f(n) ≤ c.g(n) for all
n > n0. }
2.2 Ω Notation (Big-Omega Notation)
Big-Omega, commonly written as Ω, is an Asymptotic Notation for the best case, or the best
amount of time an algorithm can possibly take to complete. It provides us with an asymptotic
lower bound for the growth rate of run-time of an algorithm.

For example, for a function f(n)

Ω(f(n)) ≥ { g(n) : there exists c > 0 and n0 such that g(n) ≤ c.f(n) for all
n > n0. }

2.3 θ Notation (Theta Notation)

Theta, commonly written as Θ, is an Asymptotic Notation to denote the asymptotically tight
bound (lower bound and the upper bound) on the growth rate of run-time of an algorithm.

For example, for a function f(n)

θ(f(n)) = { g(n) if and only if g(n) = Ο(f(n)) and g(n) = Ω(f(n)) for all n
> n0. }
3. Common Asymptotic Notations
Following is a list of some common asymptotic notations −

Notation Name

O(1) constant

O(log n) logarithmic

O(n) linear

O(n log n) = O(log n!) linearithmic, loglinear, or quasilinear

O(n2) quadratic

O(nc) polynomial or algebraic

O(cn); where c>1 exponential

O(n!) factorial

Big-O Notation Explained with Examples

Asymptotic notation is a set of languages which allow us to express the performance of our
algorithms in relation to their input. Big O notation is used in Computer Science to describe the
performance or complexity of an algorithm. Big O specifically describes the worst-case
scenario, and can be used to describe the execution time required or the space used (e.g. in
memory or on disk) by an algorithm.

Big O complexity can be visualized with this graph:

Here the best way to understand Big O thoroughly examples in code. So, below are some
common orders of growth along with descriptions and examples where possible.
1. O(1)
void printFirstElementOfArray(int arr[])
{
printf("First element of array = %d",arr[0]);
}

This function runs in O(1) time (or "constant time") relative to its input. The input array could
be 1 item or 1,000 items, but this function would still just require one step.

2. O(n)
void printAllElementOfArray(int arr[], int size)
{
for (int i = 0; i < size; i++)
{
printf("%d\n", arr[i]);
}
}

This function runs in O(n) time (or "linear time"), where n is the number of items in the array.
If the array has 10 items, we have to print 10 times. If it has 1000 items, we have to print 1000
times.

3. O(n2)
void printAllPossibleOrderedPairs(int arr[], int size)
{
for (int i = 0; i < size; i++)
{
for (int j = 0; j < size; j++)
{
printf("%d = %d\n", arr[i], arr[j]);
}
}
}

Here we're nesting two loops. If our array has n items, our outer loop runs n times and our inner
loop runs n times for each iteration of the outer loop, giving us n2 total prints. Thus this
function runs in O(n2) time (or "quadratic time"). If the array has 10 items, we have to print 100
times. If it has 1000 items, we have to print 1000000 times.
4. O(2n)
int fibonacci(int num)
{
if (num <= 1) return num;
return fibonacci(num - 2) + fibonacci(num - 1);
}

An example of an O(2n) function is the recursive calculation of Fibonacci numbers. O(2n)

denotes an algorithm whose growth doubles with each addition to the input data set. The
growth curve of an O(2n) function is exponential - starting off very shallow, then rising
meteorically.

5. Drop the constants

When you're calculating the big O complexity of something, you just throw out the constants.
Like:

void printAllItemsTwice(int arr[], int size)

{
for (int i = 0; i < size; i++)
{
printf("%d\n", arr[i]);
}

for (int i = 0; i < size; i++)

{
printf("%d\n", arr[i]);
}
}

This is O(2n), which we just call O(n).

void printFirstItemThenFirstHalfThenSayHi100Times(int arr[], int size)

{
printf("First element of array = %d\n",arr[0]);

for (int i = 0; i < size/2; i++)

{
printf("%d\n", arr[i]);
}

for (int i = 0; i < 100; i++)

{
printf("Hi\n");
}
}

This is O(1 + n/2 + 100), which we just call O(n).

Why can we get away with this? Remember, for big O notation we're looking at what happens
as n gets arbitrarily large. As n gets really big, adding 100 or dividing by 2 has a decreasingly
significant effect.
6. Drop the less significant terms
void printAllNumbersThenAllPairSums(int arr[], int size)
{
for (int i = 0; i < size; i++)
{
printf("%d\n", arr[i]);
}

for (int i = 0; i < size; i++)

{
for (int j = 0; j < size; j++)
{
printf("%d\n", arr[i] + arr[j]);
}
}
}

Here our runtime is O(n + n2), which we just call O(n2).

Similarly:

• O(n3 + 50n2 + 10000) is O(n3)

• O((n + 30) * (n + 5)) is O(n2)

Again, we can get away with this because the less significant terms quickly become, well, less
significant as n gets big.

7. With Big-O, we're usually talking about the "worst case"

bool arrayContainsElement(int arr[], int size, int element)
{
for (int i = 0; i < size; i++)
{
if (arr[i] == element) return true;
}
return false;
}

Here we might have 100 items in our array, but the first item might be the that element, in this
case we would return in just 1 iteration of our loop.

In general we'd say this is O(n) runtime and the "worst case" part would be implied. But to be
more specific we could say this is worst case O(n) and best case O(1) runtime. For some
algorithms we can also make rigorous statements about the "average case" runtime.
8. Other Examples
Let's take the following C example which contains a for loop, iterates from i = 0 to i <
10000 and prints each value of that i:

#include<stdio.h>
void print_values(int end)
{
for (int i = 0; i < end; i++)
{
printf("%d\n", i);
}
}
int main()
{
print_values(10000);
return 0;
}

We could put a timer at the beginning and the end of the line of code which calls this function,
this would then give us the running time of our print_values algorithm, right?

#include<stdio.h>
#include<time.h>
void print_values(int end)
{
for (int i = 0; i < end; i++)
{
printf("%d\n", i);
}
}
int main()
{
clock_t t;
t = clock();

print_values(10000);

float diff = ((float)(clock() - t)) / CLOCKS_PER_SEC ;

printf ("\n\n diff=%f \n\n", diff);

return 0;
}

Maybe, but what if you run it again, three times, write down your results and then move to
another machine with a higher spec and run it another three times. I bet upon comparison of the
results you will get different running times!

This is where asymptotic notations are important. They provide us with a mathematical
foundation for representing the running time of our algorithms consistently.

We create this consistency by talking about operations our code has to perform. Operations
such as array lookups, print statements and variable assignments.

If we were to annotate print_values with the amount of times each line within the function is
executed for the input 10000, we would have something as follows:
void print_values(int end) //end = 10000
{
for (int i = 0; i < end; i++) //Execution count: 10000
{
printf("%d\n", i); // Execution count: 10000
}
}

If we were to change the input value of print_values function, our print statement would be
exercised more or less, depending on the value of that input.

If we were to put this into an arithmetic expression, we would get 10000+1, using intuition we
know that the 10000 is variable on the input size, if we call the input value n, we would now
have the expression n+1.

I could now argue that the worst case running time for print_values is O(n+1). n for the loop
block and 1 for the print statement.

In the grand scheme of things, the constant value 1 is pretty insignificant at the side of the
variable value n. So we simply reduce the above expression to O(n), and there we have our
Big-O running time of print_values.

As our code prints each and every value from 0 to the input , as the loop is the most significant
part of the code, we are able to say that our code is of running time O(n) where n is the variable
length of the array! Simples!

An algorithm of running time O(n) is said to be linear, which essentially means the algorithms
running time will increase linearly with its input (n).

9. Proving Big-O
We can prove, mathematically, that print_values is in-fact O(n), which brings us on to the
formal definition for Big-O:

f(n) = O(g(n)) if c and some initial value k are positive when f(n) <= c * g(n) for all n > k is
true.

We can turn this formal definition into an actual definition of our above code, which we can
then in turn prove.

We must first ask does print_values have a running time of O(n)?

If print_values <= c * n, when c = 1 then print_values does have a running time of

O(n) when n > k.

c can be any integer while k is the amount of iterations we must perform for the expression to
be true for every subsequent value of n.
As c is just 1, we can simplify our expression to print_values <= n.

n f(n) g(n) True/False

0 0 0 False
1 1 1 True
2 2 2 True
3 3 3 True

We can see that n must be greater than the value 0 of constant k in order to satisfy the
expression print_values <= n.

We can now say when n is 1:

1 <= 1 * 1 for 1 > 0 is true. We know this because 1 multiplied by 1 is 1 and 1 is greater than
our constant k which was 0.

The above must be true for all values of n greater than k (0), so if n was 10, 10 <= 1 * 10 for 10
> 0 is also true.

What we're basically saying here is that no matter our input (n), it must be greater than or equal
to our constant (c) when the size of our input (n) is more than another constant value (k), in our
case the iteration count of the function.

But where do our constants come from? Well they are just values, we typically start at 1 and
work our way up to seek a constant which makes the expression f(n) <= c * g(n) for all n > k
true. If we cannot find such combination of constants, then our code does not have a running
time of O(n) and our hypothesis was incorrect.

10 Disproving Big-O
Lets take a new C function, which contains a for loop, iterates from i = 0 to i < 100 and an
another nested for loop from j = 0 to j < 100 which prints each value of that i and j:

void print_values_with_repeat(int end) //end = 100

{
for (int i = 0; i < end; i++)
{
for (int j = 0; j < end; j++)
{
printf("i = %d and j = %d\n", i, j);
}
}
}
If we were to annotate print_values_with_repeat with the amount of times each line within
the function is executed for the input 100, we would have something as follows:

void print_values_with_repeat(int end) //end = 100

{
for (int i = 0; i < end; i++) //Execution count: 100
{
for (int j = 0; j < end; j++) //Execution count: 10000
{
printf("i = %d and j = %d\n", i, j); // Execution count: 1
}
}
}

Does print_values_with_repeat have a running time of O(n)?

n f(n) g(n) True/False

0 0 0 False
1 1 1 True
2 4 2 False
3 9 3 False

Suppose our constant c is 1, 1 <= 1 * 1 for 1 > 0, this is true - however our definition says that
g(n) must be greater than all values of f(n).

So if we take the value 2 of n, 2 <= 1 * 4 for 1 > 0, we can see that this is now false, which
disproves our hypothesis that print_values_with_repeat is O(n). Even if we change our
constant c to 2, this would still prove false eventually.

We can actually see that the order of growth in operations in print_values_with_repeat is

actually n2, so let's hypothesise now that print_values_with_repeat is actually O(n2).

Does print_values_with_repeat have a running time of O(n2)?

n f(n) g(n) True/False

0 0 0 False
1 1 1 True
2 4 4 True
3 9 9 True

Suppose our constant c is still 1, our expression would now be 3 <= 3 * 32 for 3 > 0, this is true,
great! print_values_with_repeat is in-fact O(n2).

O(n2) is a quadratic time algorithm, as the running time of the algorithm increases quadratically
to the input.
Asymptotic Notation 1

Growth of Functions and

Aymptotic Notation
• When we study algorithms, we are interested in
characterizing them according to their efficiency.
• We are usually interesting in the order of growth
of the running time of an algorithm, not in the
exact running time. This is also referred to as the
asymptotic running time.
• We need to develop a way to talk about rate of
growth of functions so that we can compare
algorithms.
• Asymptotic notation gives us a method for
classifying functions according to their rate of
growth.
Asymptotic Notation 2

Big-O Notation
• Definition: f (n) = O(g(n)) iff there are two
positive constants c and n0 such that
|f (n)| ≤ c |g(n)| for all n ≥ n0
• If f (n) is nonnegative, we can simplify the last
condition to
0 ≤ f (n) ≤ c g(n) for all n ≥ n0
• We say that “f (n) is big-O of g(n).”
• As n increases, f (n) grows no faster than g(n).
In other words, g(n) is an asymptotic upper
bound on f (n).

cg(n)
f(n)

f(n) = O(g(n))

n0
Asymptotic Notation 3

Example: n2 + n = O(n3 )

Proof:
• Here, we have f (n) = n2 + n, and g(n) = n3
• Notice that if n ≥ 1, n ≤ n3 is clear.
• Also, notice that if n ≥ 1, n2 ≤ n3 is clear.
• Side Note: In general, if a ≤ b, then na ≤ nb
whenever n ≥ 1. This fact is used often in these
types of proofs.
• Therefore,
n2 + n ≤ n3 + n3 = 2n3

• We have just shown that

n2 + n ≤ 2n3 for all n ≥ 1

• Thus, we have shown that n2 + n = O(n3 )

(by definition of Big-O, with n0 = 1, and c = 2.)
Asymptotic Notation 4

Big-Ω notation
• Definition: f (n) = Ω(g(n)) iff there are two
positive constants c and n0 such that
|f (n)| ≥ c |g(n)| for all n ≥ n0
• If f (n) is nonnegative, we can simplify the last
condition to
0 ≤ c g(n) ≤ f (n) for all n ≥ n0
• We say that “f (n) is omega of g(n).”
• As n increases, f (n) grows no slower than g(n).
In other words, g(n) is an asymptotic lower bound
on f (n).

f(n)

cg(n)

f(n) = O(g(n))

n0
Asymptotic Notation 5

Example: n3 + 4n2 = Ω(n2 )

Proof:
• Here, we have f (n) = n3 + 4n2 , and g(n) = n2
• It is not too hard to see that if n ≥ 0,
n3 ≤ n3 + 4n2

• We have already seen that if n ≥ 1,

n2 ≤ n3

• Thus when n ≥ 1,
n2 ≤ n3 ≤ n3 + 4n2

• Therefore,
1n2 ≤ n3 + 4n2 for all n ≥ 1

• Thus, we have shown that n3 + 4n2 = Ω(n2 )

(by definition of Big-Ω, with n0 = 1, and c = 1.)
Asymptotic Notation 6

Big-Θ notation
• Definition: f (n) = Θ(g(n)) iff there are three
positive constants c1 , c2 and n0 such that
c1 |g(n)| ≤ |f (n)| ≤ c2 |g(n)| for all n ≥ n0
• If f (n) is nonnegative, we can simplify the last
condition to
0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n) for all n ≥ n0
• We say that “f (n) is theta of g(n).”
• As n increases, f (n) grows at the same rate as
g(n). In other words, g(n) is an asymptotically
tight bound on f (n).

c2 g(n) f(n)

c1 g(n)

n0
Asymptotic Notation 7

Example: n2 + 5n + 7 = Θ(n2 )

Proof:
• When n ≥ 1,
n2 + 5n + 7 ≤ n2 + 5n2 + 7n2 ≤ 13n2

• When n ≥ 0,
n2 ≤ n2 + 5n + 7

• Thus, when n ≥ 1
1n2 ≤ n2 + 5n + 7 ≤ 13n2
Thus, we have shown that n2 + 5n + 7 = Θ(n2 )
(by definition of Big-Θ, with n0 = 1, c1 = 1, and
c2 = 13.)
Asymptotic Notation 8

Arithmetic of Big-O, Ω, and Θ notations

• Transitivity:
– f (n) ∈ O(g(n)) and
g(n) ∈ O(h(n)) ⇒ f (n) ∈ O(h(n))
– f (n) ∈ Θ(g(n)) and
g(n) ∈ Θ(h(n)) ⇒ f (n) ∈ Θ(h(n))
– f (n) ∈ Ω(g(n)) and
g(n) ∈ Ω(h(n)) ⇒ f (n) ∈ Ω(h(n))
• Scaling: if f (n) ∈ O(g(n)) then for any
k > 0, f (n) ∈ O(kg(n))
• Sums: if f1 (n) ∈ O(g1 (n)) and
f2 (n) ∈ O(g2 (n)) then
(f1 + f2 )(n) ∈ O(max(g1 (n), g2 (n)))
Asymptotic Notation 9

Strategies for Big-O

• Sometimes the easiest way to prove that
f (n) = O(g(n)) is to take c to be the sum of the
positive coefficients of f (n).
• We can usually ignore the negative coefficients.
Why?
• Example: To prove 5n2 + 3n + 20 = O(n2 ), we
pick c = 5 + 3 + 20 = 28. Then if n ≥ n0 = 1,
5 n2 + 3 n + 20 ≤ 5 n2 + 3 n2 + 20 n2 = 28 n2 ,
thus 5n2 + 3n + 20 = O(n2 ).
• This is
√notlogalways so easy. How would you show
that ( 2) n + log2 n + n4 is O(2n )? Or that
n2 = O(n2 − 13n + 23)? After we have talked
about the relative rates of growth of several
functions, this will be easier.
• In general, we simply (or, in some cases, with
much effort) find values c and n0 that work. This
gets easier with practice.
Asymptotic Notation 10

Strategies for Ω and Θ

• Proving that a f (n) = Ω(g(n)) often requires
more thought.
– Quite often, we have to pick c < 1.
– A good strategy is to pick a value of c which
you think will work, and determine which
value of n0 is needed.
– Being able to do a little algebra helps.
– We can sometimes simplify by ignoring terms
of f (n) with the positive coefficients. Why?
• The following theorem shows us that proving
f (n) = Θ(g(n)) is nothing new:
– Theorem: f (n) = Θ(g(n)) if and only if
f (n) = O(g(n)) and f (n) = Ω(g(n)).
– Thus, we just apply the previous two
strategies.
• We will present a few more examples using a
several different approaches.
Asymptotic Notation 11

Show that 12 n2 + 3n = Θ(n2 )

Proof:
• Notice that if n ≥ 1,
1 2 1 7
n + 3n ≤ n2 + 3n2 = n2
2 2 2
• Thus,
1 2
n + 3n = O(n2 )
2
• Also, when n ≥ 0,
1 2 1 2
n ≤ n + 3n
2 2
• So
1 2
n + 3n = Ω(n2 )
2
• Since 21 n2 + 3n = O(n2 ) and 21 n2 + 3n = Ω(n2 ),
1 2
n + 3n = Θ(n2 )
2
Asymptotic Notation 12

Show that (n log n − 2 n + 13) = Ω(n log n)

Proof: We need to show that there exist positive

constants c and n0 such that
0 ≤ c n log n ≤ n log n − 2 n + 13 for all n ≥ n0 .
Since n log n − 2 n ≤ n log n − 2 n + 13,
we will instead show that
c n log n ≤ n log n − 2 n,
which is equivalent to
2
c≤1− , when n > 1.
log n
If n ≥ 8, then 2/(log n) ≤ 2/3, and picking c = 1/3
suffices. Thus if c = 1/3 and n0 = 8, then for all
n ≥ n0 , we have
0 ≤ c n log n ≤ n log n − 2 n ≤ n log n − 2 n + 13.
Thus (n log n − 2 n + 13) = Ω(n log n).
Asymptotic Notation 13

Show that 12 n2 − 3n = Θ(n2 )

Proof:
• We need to find positive constants c1 , c2 , and n0
such that
1 2
0 ≤ c1 n2 ≤ n − 3 n ≤ c2 n2 for all n ≥ n0
2
• Dividing by n2 , we get
1 3
0 ≤ c1 ≤ − ≤ c2
2 n
1 3
• c1 ≤ 2 − n holds for n ≥ 10 and c1 = 1/5
1 3
• 2 − n ≤ c2 holds for n ≥ 10 and c2 = 1.
• Thus, if c1 = 1/5, c2 = 1, and n0 = 10, then for
all n ≥ n0 ,
1 2
0 ≤ c1 n2 ≤ n − 3 n ≤ c2 n2 for all n ≥ n0 .
2
Thus we have shown that 12 n2 − 3n = Θ(n2 ).
Asymptotic Notation 14

Asymptotic Bounds and Algorithms

• In all of the examples so far, we have assumed we
knew the exact running time of the algorithm.
• In general, it may be very difficult to determine
the exact running time.
• Thus, we will try to determine a bounds without
computing the exact running time.
• Example: What is the complexity of the
following algorithm?
for (i = 0; i < n; i ++)
for (j = 0; j < n; j ++)
a[i][j] = b[i][j] * x;
Answer: O(n2 )
• We will see more examples later.
Asymptotic Notation 15

Summary of the Notation

• f (n) ∈ O(g(n)) ⇒ f  g
• f (n) ∈ Ω(g(n)) ⇒ f  g
• f (n) ∈ Θ(g(n)) ⇒ f ≈ g
• It is important to remember that a Big-O bound is
only an upper bound. So an algorithm that is
O(n2 ) might not ever take that much time. It may
actually run in O(n) time.
• Conversely, an Ω bound is only a lower bound. So
an algorithm that is Ω(n log n) might actually be
Θ(2n ).
• Unlike the other bounds, a Θ-bound is precise. So,
if an algorithm is Θ(n2 ), it runs in quadratic time.
Asymptotic Notation 16

Common Rates of Growth

In order for us to compare the efficiency of algorithms,

we need to know some common growth rates, and how
they compare to one another. This is the goal of the
next several slides.
Let n be the size of input to an algorithm, and k some
constant. The following are common rates of growth.
• Constant: Θ(k), for example Θ(1)
• Linear: Θ(n)
• Logarithmic: Θ(logk n)
• n log n: Θ(n logk n)
• Quadratic: Θ(n2 )
• Polynomial: Θ(nk )
• Exponential: Θ(k n )
We’ll take a closer look at each of these classes.
Asymptotic Notation 17

Classification of algorithms - Θ(1)

• Operations are performed k times, where k is
some constant, independent of the size of the
input n.
• This is the best one can hope for, and most often
unattainable.
• Examples:
int Fifth_Element(int A[],int n) {
return A[5];
}

int Partial_Sum(int A[],int n) {

int sum=0;
for(int i=0;i<42;i++)
sum=sum+A[i];
return sum;
}
Asymptotic Notation 18

Classification of algorithms - Θ(n)

• Running time is linear
• As n increases, run time increases in proportion
• Algorithms that attain this look at each of the n
inputs at most some constant k times.
• Examples:
void sum_first_n(int n) {
int i,sum=0;
for (i=1;i<=n;i++)
sum = sum + i;
}
void m_sum_first_n(int n) {
int i,k,sum=0;
for (i=1;i<=n;i++)
for (k=1;k<7;k++)
sum = sum + i;
}
Asymptotic Notation 19

Classification of algorithms - Θ(log n)

• A logarithmic function is the inverse of an
exponential function, i.e. bx = n is equivalent to
x = logb n)
• Always increases, but at a slower rate as n
increases. (Recall that the derivative of log n is n1 ,
a decreasing function.)
• Typically found where the algorithm can
systematically ignore fractions of the input.
• Examples:
int binarysearch(int a[], int n, int val)
{
int l=1, r=n, m;
while (r>=1) {
m = (l+r)/2;
if (a[m]==val) return m;
if (a[m]>val) r=m-1;
else l=m+1; }
return -1;
}
Asymptotic Notation 20

Classification of algorithms - Θ(n log n)

• Combination of O(n) and O(log n)
• Found in algorithms where the input is recursively
broken up into a constant number of subproblems
of the same type which can be solved
independently of one another, followed by
recombining the sub-solutions.
• Example: Quicksort is O(n log n).

Perhaps now is a good time for a reminder that when

speaking asymptotically, the base of logarithms is
irrelevant. This is because of the identity
loga b logb n = loga n.
Asymptotic Notation 21

Classification of algorithms - Θ(n2 )

• We call this class quadratic.
• As n doubles, run-time quadruples.
• However, it is still polynomial, which we consider
to be good.
• Typically found where algorithms deal with all
pairs of data.
• Example:
int *compute_sums(int A[], int n) {
int M[n][n];
int i,j;
for (i=0;i<n;i++)
for (j=0;j<n;j++)
M[i][j]=A[i]+A[j];
return M;
}
• More generally, if an algorithm is Θ(nk ) for
constant k it is called a polynomial-time
algorithm.
Asymptotic Notation 22

Classification of algorithms - Θ(2n )

• We call this class exponential.
• This class is, essentially, as bad as it gets.
• Algorithms that use brute force are often in this
class.
• Can be used only for small values of n in practice.
• Example: A simple way to determine all n bit
numbers whose binary representation has k
non-zero bits is to run through all the numbers
from 1 to 2n , incrementing a counter when a
number has k nonzero bits. It is clear this is
exponential in n.
Asymptotic Notation 23

Comparison of growth rates

log n n n log n n2 n3 2n
0 1 0 1 1 2
0.6931 2 1.39 4 8 4
1.099 3 3.30 9 27 8
1.386 4 5.55 16 64 16
1.609 5 8.05 25 125 32
1.792 6 10.75 36 216 64
1.946 7 13.62 49 343 128
2.079 8 16.64 64 512 256
2.197 9 19.78 81 729 512
2.303 10 23.03 100 1000 1024
2.398 11 26.38 121 1331 2048
2.485 12 29.82 144 1728 4096
2.565 13 33.34 169 2197 8192
2.639 14 36.95 196 2744 16384
2.708 15 40.62 225 3375 32768
2.773 16 44.36 256 4096 65536
2.833 17 48.16 289 4913 131072
2.890 18 52.03 324 5832 262144
log log m log m m
Asymptotic Notation 24

More growth rates

n 100n n2 11n2 n3 2n
1 100 1 11 1 2
2 200 4 44 8 4
3 300 9 99 27 8
4 400 16 176 64 16
5 500 25 275 125 32
6 600 36 396 216 64
7 700 49 539 343 128
8 800 64 704 512 256
9 900 81 891 729 512
10 1000 100 1100 1000 1024
11 1100 121 1331 1331 2048
12 1200 144 1584 1728 4096
13 1300 169 1859 2197 8192
14 1400 196 2156 2744 16384
15 1500 225 2475 3375 32768
16 1600 256 2816 4096 65536
17 1700 289 3179 4913 131072
18 1800 324 3564 5832 262144
19 1900 361 3971 6859 524288
Asymptotic Notation 25

More growth rates

n n2 n2 − n n2 + 99 n3 n3 + 234
2 4 2 103 8 242
6 36 30 135 216 450
10 100 90 199 1000 1234
14 196 182 295 2744 2978
18 324 306 423 5832 6066
22 484 462 583 10648 10882
26 676 650 775 17576 17810
30 900 870 999 27000 27234
34 1156 1122 1255 39304 39538
38 1444 1406 1543 54872 55106
42 1764 1722 1863 74088 74322
46 2116 2070 2215 97336 97570
50 2500 2450 2599 125000 125234
54 2916 2862 3015 157464 157698
58 3364 3306 3463 195112 195346
62 3844 3782 3943 238328 238562
66 4356 4290 4455 287496 287730
70 4900 4830 4999 343000 343234
74 5476 5402 5575 405224 405458
 Asymptotic Notation
Polynomial Functions
40000
x
35000 x**2
x**3
30000 x**4
25000
20000
15000
10000
5000
0
0 5 10 15 20 25 30 35 40

26
 Asymptotic Notation
Slow Growing Functions
250
log(x)
x
200 x*log(x)
x**2
150

100

50

0
0 5 10 15 20 25 30 35 40

27
 Asymptotic Notation
Fast Growing Functions Part 1
5000
x
4500 x**3
4000 x**4
2**x
3500
3000
2500
2000
1500
1000
500
0
0 2 4 6 8 10

28
 Asymptotic Notation
Fast Growing Functions Part 2
500000
x
450000 x**3
400000 x**4
2**x
350000
300000
250000
200000
150000
100000
50000
0
0 5 10 15 20

29
 Asymptotic Notation
Why Constants and Non-Leading Terms Don’t Matter
4e+08
1000000*x
3.5e+08 300000*x**2 + 300*x
2**x
3e+08
2.5e+08
2e+08
1.5e+08
1e+08
5e+07
0
0 5 10 15 20 25 30

30
Asymptotic Notation 31

Classification Summary

We have seen that when we analyze functions

asymptotically:
• Only the leading term is important.
• Constants don’t make a significant difference.
• The following inequalities hold asymptotically:

2
√
c < log n < log n < n < n < n log n

n < n log n < n(1.1) < n2 < n3 < n4 < 2n

• In other words, an algorithm that is Θ(n log(n)) is
more efficient than an algorithm that is Θ(n3 ).