1 Python Programming

The previous Chapter provided an overview of modeling in optimization. The

problems considered were intentionally simple, and could therefore be solved
analytically. In practice, most optimization problems in engineering entail
numerical analysis and software programming.
In this Chapter, we discuss a particular software language, namely Python, for
developing various optimization algorithms. The reasons for choosing Python,
over, say, Matlab or C++, are:
(1) Python is free
(2) It is object-oriented
(3) It is interpreted
(4) It is operating-system independent
(5) It has an excellent optimization module
(6) It offers modern COM modules for interfacing with SolidsWorks.
In short, it is an ideal programming language for students and practicing
engineers to learn optimization. In this Chapter, we highlight some of the salient
features of Python. A good grasp of these fundamentals is sufficient for the
remainder of this text. However, the reader is encouraged to consult
www.python.org, or other Python texts [References], as needed.
1.1 Getting Started with Python
The easiest way to get started with Python is to download the Python(x,y)
package from http://code.google.com/p/pythonxy/. The Python(x,y) package
comes with all numerical and scientific Python modules that we will need.
Further, included in Python(x,y) is Spyder, an excellent integrated development
environment (IDE).
Once Python(x,y) has been installed, fire-up Spyder, and the following set of
windows should show up. Then, in the Console window, type print Hello
World! and hit enter. If everything goes well, the response should be as shown.
(Note that your particular configuration may be slightly different). For those of
you familiar with Matlab, you will find it easy to use Spyder IDE.
You can configure the Spyder IDE through Tools-Preferences Menu. For
example, you can add/remove line numbers on the editor window, increase font
size, change colors etc. You can also move the file-explorer and console windows,
by directly dragging/dropping them to the desired location.
The reader should be aware that Spyder can run under two Console modes,
namely, the standard Python mode, and the more powerful Interactive Python
(IPython) mode. We will use the IPython mode for the remainder of this text. You
can choose the appropriate mode under the Run menu.

Figure 1-1: The Spyder IDE.
1.2 Basic Objects
Python is an object-oriented language. Indeed, everything one creates in
Python is an object, including integers, float, strings, arrays, etc. Below are a few
basic objects, assigned them to variables i', x and a.

Associated with objects are methods that act on these objects. By typing a dot
after the object variable name, you can access a list of methods associated with it.
For example, with the string object a are methods to capitalize the first letter,
find a substring, etc. This is illustrated below; the reader is encouraged to
familiarize themselves with various methods associated with integer, float and
string objects.

One can manipulate, combine and display objects as illustrated below. Observe
that the meaning of the + operator is context dependent. For integers and floats,
it is interpreted as the usual addition; for strings, it is interpreted in Python as a
concatenation.

Also, observe that, one can assign new variables to existing objects (as in b = a,
below). Reassigning the variable b to a different object (in this case) does not
alter the contents of a (as in b = world!).

1.3 Lists
A particularly useful object in Python is that of a list that is simply a collection
of other Python objects. Below are a few examples of list objects. Observe that: (1)
lists can contain a variety of objects (integers, strings, etc), and (2) they can even
contain other list objects as in b = [34, a], (3) addition of lists leads to a
concatenation as in c = a + a, and (4) access to individual elements of a list is
through the [] operator (as in a[2]); the indexing of individual elements of a list
starts from 0, rather than 1.

List objects are however different from integers, floats and strings in the
following sense. In the example below, observe that a and b variables are
initially pointing to the same list object. However, modifying b, by accessing its
individual elements, modifies the underlying list, and is therefore reflected in the
variable a. This fundamental difference is to be noted.

A particularly useful method of creating a list of integers is the range function.

Lists play an important role in this text since arrays (to be discussed later on) are
a special incarnation of lists.
1.4 Python Scripts
The Console window, which we used in the previous examples, plays an
important role for interactive computing. However, to develop larger programs,
we rely on the Editor within Spyder. For example, we have created a simple
Python program in the Editor, and saved the file as 2.1-PythonObjects.py, where
.py extension refers to a Python file.

You can execute the above program through the Run command (or via the F5
short-cut). When the Python program is executed for the first time in Spyder, the
following configuration window opens up. In the window, choose the Interactive
Python (IPython) mode, and hit Run.

Then, the following output appears in the Console window.

The reader is encouraged to study the Python script and the output.
In the above script file, we have included two types of Python comments: (1)
the hash (#) symbol used for single-line comments, i.e., the remainder of the
line following the # is treated as a comment, and (2) the triple-quotes for multi-
line comments, i.e., any text between two triple-quotes is treated as a comment;
the triple-quotes also serves as a means of documenting, also referred to as a
docstring, the script file (we shall revisit this topic later on).
1.5 Flow Control
As in any programming language, Python offers a variety of options for flow
control. The following example illustrates the use of for, if and while
commands in Python

Observe that there is no end statement that identifies the end of the for-block.
Indeed, in Python, the end is implied by the consistent use of indentation. The
reader is encouraged to mess-around with the indentation and study the impact
on the output. The output of the above script is shown below.

1.6 User Input
Python provides two commands, namely raw_input and input for user
interaction. The first command returns the user input as a string, while the
second command will interpret and evaluate the input, and return the interpreted
value (say a float or a list), if the evaluation is meaningful. The difference between
the two commands is illustrated below.

1.7 Numerical Python
Thus far, we have focused on the core Python language. However, there are
other numerical objects (such as arrays) and methods (such as the dot product)
that are not part of the core Python language, but are part of the numpy and scipy
libraries/modules. These libraries have been installed on your machine when you
installed Python(x,y). However, in order to access them in a script file, you must
import them (these libraries are automatically imported in the Console window.)
We can import each of these modules individually via import numpy, import
scipy, etc. Alternately, all popular Python modules may be imported via the
command import pylab as in line-2 below. Note that we have used the
abbreviation py as a synonym for pylab.
Various constants (such as ), objects (such as an array), and methods (such
as sin, cosine) can now be accessed via Pylab, as illustrated below. The reader is
encouraged to experiment with these and other methods within Pylab.

The resulting output in the Console window is shown below.

1.8 Linear Algebra
Pylab also supports numerous matrix operations; below are a few examples.

Observe, in the above script, that:
The solve function solves Ax = b
The eigen values of a matrix are computed via the eig function
The dot function performs the operation A*x
norm is the usual 2-norm of a vector, defined as
( ) ( ) ( )
2 2 2
0 1 1
...
N
x x x

= x
The output is shown below.

1.9 Complex Numbers
Python also supports the use of complex numbers through the use of symbol
j that represents 1 . Below are a few examples to illustrate the use of complex
numbers.

1.10 Plots
The ability to plot and visualize functions is extremely valuable in
optimization. Pylab supports 2-D and 3-D plotting via matlibplot
(http://matplotlib.sourceforge.net) package that can be accessed through pylab.
Below is a simple example to illustrate 2-D plotting of functions (numerous other
examples can be found at the matplotlib web-site).

This results in the figure below; the reader is encouraged to experiment with the
code.

Figure 1-2: Plot of sin( ) x and cos( ) x .
Matlibplot offers various 3-D plotting routines as well; below is an example to
plot the function:

2 4
( , ) ( 1) ( 4) f = (1.1)

This results in the plot shown below.

Figure 1-3: Surface plot of
2 4
( , ) ( 1) ( 4) f =
An alternate means of visualizing 2-dimensional functions is through a contour
plot. A contour is the set of (x, y) points for which the function ( , ) f x y takes a
constant value, i.e.,

2
0 0
( , ) ( , ) | ( , ) C f c x y R f x y c = = (1.2)
We will study the importance of contours in later Chapters. Continuing with the
previous script, the following script shows two ways to plot contours using
matlibplot.

This results in 2 different contour plots shown below.

Figure 1-4: 3-D and 2-D contour plot of
2 4
( , ) ( 1) ( 4) f = .
Note that:
1. There are infinite contours for a function, but no two contours may cross
each other
2. For every point
0 0
( , ) x y there exists a unique contour passing through that
point, however, for a given value of the right hand side value
0
c in
Equation, a contour may not exist, or there may be multiple contours.
3. At every point
0 0
( , ) x y , the gradient of the function, i.e., f (a 2-
dimensional vector), is perpendicular to the contour passing through that
point, and its direction is along increasing contour values of f .
1.11 User-Defined Functions
Thus far, we have considered built-in Python functions from existing
modules. One can, of course, create custom functions. For example, consider the
function

2
( ) 1
x
f x xe

= (1.3)
One can create the above function via the keyword def in Python as shown
below. The script is saved in a file called myFunction.py.

Once the above file has been created, you can load and use the function as follows
(assuming that the above file is in the current directory of Spyder).

One can now also use the plot function as follows (observe the vectorized
operation):

This results in the figure below.

Figure 1-5: Plot of
2
1
x
xe

.
1.12 Modules
One can include multiple functions within a single Python file, and access
each one of them individually (a distinct advantage over Matlab). For example,
below is a file SimpleFunctions.py containing multiple functions. For example,
the function f3 implements

2 4
( , ) ( 1) ( 4) f = (1.4)


Each of these functions is accessible once the SimpleFunctions.py has been
imported, as illustrated below.

Further, one can query the above module via the help command in Python:

Observe that Python has intelligently interpreted your code via the triple-quotes,
and provided a useful synopsis.
1.13 Module Testing
One can test individual modules (such as the SimpleFunctions.py) by
including a main function as part of the file. For example, continuing with the
above script, we add the following lines of Python code (observe the special
syntax of line-34).

One can now run the SimpleFunctions.py module (through F5 shortcut), and the
result is shown below.

Figure 1-6: Plot of
2
1
x
xe

, and its gradient.

1.14 Function Arguments
Python offers a rich set of language features for passing arguments into
functions. For example, consider the function f1 below (together with a testing
script). The function takes up to 3 arguments: the argument x does not have a
default value, while the argument a takes the default value of 4, and s takes the
default value of hello. Any variable that does not have a default value must be
specified when calling functions.
In any call to a function, arguments can be passed either by position, f1(0.3),
or keyword, f1(x=0.3). This allows for a highly flexible and natural way for
passing data to a function. When passing by position, one must respect the
position defined in the function description, but when passing by keyword any
order is acceptable. In addition, when mixing position and keyword styles, all
positional arguments must come before keyword arguments.

The result of running the above script is displayed below.

1.15 Python Quirks
There are a few Python quirks that one must keep in mind. Consider the
following division of two real numbers in Python; the result is as expected

Now consider the following.

Python returns the integer component of the result. This implicit integer
interpretation is common to many languages, and can cause headaches in
numerical computation; the reader should therefore be very watchful of implicit
integer conversion. One can override this behavior as follows

In Python modules, the import statement must appear at the first line.
Henceforth, we shall include the above statement in all Python modules.
A second feature unique to Python is the strict adherence to indentation at
the beginning of an expression. Consider the following code:

The only difference between this and the previous code was the inclusion of an
extra space before the number of 5.0. Python does not allow extraneous use of
space or tabs at the beginning of expressions. The underlying reason is that it
uses spaces and tabs to identify blocks of code associated. We shall revisit this
concept again later on.
The third aspect is particularly relevant to readers familiar with Matlab.
Consider the multplication of a 2x2 matrix and a 2x1 vector. In Matlab, one
would construct a matrix and vector as follows:

Then, the multiplication in Matlab is as follows:

Mathematically, in Matlab, the '*' operator is interpreted as:

i ij j
j
b A x =

(1.5)
In Python, the syntax for creating matrices and vectors is a bit more elaborate:

Now consider the following multiplication in Python:

Unfortunately, this creates a matrix b:

In Python, the '*' operator is interpreted as:

ij ij j
b A x = (1.6)
The desired matrix-vector multiplication of Equation (1.5) is achieved in Python
through the 'dot' method:

The reader has been warned!
1.16 Sampling Algorithm
As an illustrative example of a Python program, consider an algorithm that
finds the global minimum of a one-dimensional function ( ) f x within a given
interval [ , ] a b through uniform sampling.
The Python script below contains the function Sampling that takes as
arguments 3 parameters: (1) a function name that must be minimized, (2) an
interval to search within, and (3) number of sampling points.

Its usage is as follows.

We shall compare the performance of this algorithm against more sophisticated
optimization algorithms later on.
1.17 Newton-Raphson Algorithm
The Newton-Raphson algorithm is used extensively to solve non-linear
equations of the form:
( ) 0 = g x (1.7)
The theory behind Newton-Raphson algorithm is discussed later in the text, but
briefly, the algorithm is iterative in that given an initial guess
0
x , it typically finds
increasingly better solutions via:

1
1
( ) ( )
i i i i

l
=
l
x x J x g x (1.8)
where ( ) J x is the Jacobian matrix, associated with Equation (1.7), defined as:

i
ij
j
g
x

J (1.9)
Example 1-1: Find a solution to the set of nonlinear equations:

2 2
2 2 4 0
2 2 0
4 3 0
x x
y y
x x y

=
=
=
(1.10)
Solution: Note that the 3 unknowns are { , , } x y = x . As the reader can verify,
the Jacobian is given by:

2 2 0 2 4
0 2 2 2
2 4 2 0
x
y
x y

l

l
l
=
l
l
l

l
l
J (1.11)
Let
0
{0, 0, 0}
T
= x ; thus

0
0
( ) 0
3
' '
1 1
1 1
1 1
1 1
1 1
=
! !
1 1
1 1
1 1
1 1
1 1 + +
g x and
0
2 0 4
( ) 0 2 0
4 0 0
l

l
l
=
l
l
l

l
l
J x (1.12)
From Equation (1.8)

1
0.75 0 0.375
T
= x , and

1
0.5625
( ) 0
0.5625
' '
1 1
1 1
1 1
1 1
1 1
=
! !
1 1
1 1
1 1
1 1
1 1 + +
g x and
1
2.75 0 2.5
( ) 0 2.75 0
2.5 0 0
l

l
l
=
l
l
l

l
l
J x (1.13)
leading to

1
0.975 0 0.8475
T
= x . After few more iterations, the solution
converges towards

*
1 0 1
T
= x , the exact solution.

In practice, it is challenging to compute the Jacobian analytically via Equation
(1.9). Instead, a finite difference approximation may be used:

( ) ( )
i j i
ij
g he g
h

x x
J (1.14)
where
1 2
{1, 0, .., 0} , {0,1, .., 0}
T T
e e = = , etc.
We implement the Newton-Raphson method via a Python method 'fsolve'
(mimicking the Matlab function with the same name). The first few lines of the
fsolve module is as follows:

Observe that the function definition has two essential arguments: (1) the function
( ) g x and (2) the initial guess
0
x . Optional keyword arguments include the
Jacobian function, tolerances for termination and the maximum allowed
iterations.
If the Jacobian function is not provided, we rely on equation (1.14) to compute
an approximate Jacobian; its implementation is as follows:

Observe that the above function is defined inside the fsolve function. Then, the
main Newton-Raphson iteration is implemented as follows:

The reader is encouraged to study the above Python implementation.
The 'fsolve' function may be used as follows . First the g function is defined
per Equation (1.10):

One can now solve the nonlinear equations via:

Optionally, the Jacobian of Equation (1.11) may be defined via:

With this additional function, one can solve the problem via:

In both cases, the solution converges to

*
1 0 1
T
= x .
1.18 Dynamic Function Synthesis
Sometimes there is a need to dynamically synthesize or create a function. In
other words, the nature of function is determined during run-time. We will come
across such a scenario in Chapter 4. Here, we provide an illustrative example of
dynamic function synthesis.
Consider a scenario where a function ( ) f x is defined as follows:

sin( ); 0
( ; )
cos( ); 0
x
f x
x

' '
1 1
1 1
1 1
=
! !
1 1 <
1 1
1 1 + +
(1.15)
In other words, depending on the value of , the function is defined dynamically.
In Python one would implement and test this as follows.

Observe that the function fCreate returns a function object. The function
returned by fCreate depends on the auxiliary variable . Once the function is
created, it can be used like any function in Python.
1.19 Logging in Python
Python also provides excellent tools for debugging large software projects.
One such tool is logging. Below is an example to illustrate the use of logging in
Python. Observe that we have imported the logging module. There are various
methods for configuring logging; a few different options are illustrated below. In
particular, the level argument can be used to determine the behavior of the
logging utility. We shall make extensive use of logging later in the text.

When the above is executed, we see the following output on the console.

Further a file example.log is also created, with the following contents:

1.20 Python Class
Finally, we discuss the important concept of a class. A class, in object
oriented languages such as Python, is a collection of objects and methods that are
closely related. We illustrate the notion of a class through a simple example of a
polynomial class. The objective is to treat polynomials of the form:

2 1
0 1 2 1
( ) ...
N
N
p x a a x a x a x

= (1.16)
as objects with intrinsic methods such as evaluation of polynomials at a point,
addition of polynomials, etc. The polynomial in Equation (1.16) is fully defined by
the array of coefficients:

0 1 2 1
...
N
a a a a

= a (1.17)
For example, the following polynomial:

2
( ) 1 3.2 p x x x = (1.18)
is captured by the coefficients:
1, 1, 3.2 = a (1.19)
In Python, a Polynomial class may be defined via the following syntax:

Observe the use of the keyword class, and the keyword self that allows access to
quantities associated with a polynomial object. The py.array is used to copy aIn
so we do not experience the quirk we saw when dealing with lists. Once the above
class is defined in a file PolynomialClass.py, we can create a polynomial object as
follows:

The above code, by convention of the class, creates the polynomial object
representing:

2
( ) 1 3.2 p x x x = (1.20)
The class does nothing more at this point, since we have not associated any
behavior with this object. Let us now overload the __str__ method so that
print provides a human readable form:

The above code essentially retrieves the coefficients and creates a string for
display; the reader is encouraged to understand the logic behind the code. We
can now display the polynomial object created earlier as follows:

We can also add a method to evaluate polynomial objects at a particular point:

Its usage is:

The reader can verify that the reported value is indeed correct.
Next, we consider the addition of polynomials of the same order. Recall that
the addition operator takes on different interpretation depending on the object
associated with it. For integers and floats, it is interpreted as the usual addition,
while for strings and lists, it is interpreted as concatenation. We now overload the
addition operator for the polynomial class, with the natural interpretation:

( ) ( ) ( ) ( )
2 1
0 1 2 1
2 1
0 1 2 1
2 1
0 0 1 1 2 2 1 1
( ) ...
( ) ...
( ) ( ) ...
N
N
N
N
N
N N
p x a a x a x a x
q x b b x b x b x
p x q x a b a b x a b x a b x

=
=

=
(1.21)
In the Polynomial class, we create a new method with the keyword __add__
that overloads the + operator:

Its usage is:

We can continue to add other methods (such as subtraction of polynomials,
multiplication of a polynomial by a scalar, etc).
1.21 Exercises
Exercise 1-1: Under Python command prompt, evaluate the following
expressions: (a)
3 3
10 1 10 , (b)
16 16
10 1 10 , (c)
5
1 10 1

, and (d)
16
1 10 1

. Explain the apparent discrepancies.

Exercise 1-2: Explain the results of the following Python script

Exercise 1-3: Solve, using Python the 3 linear algebra problems:

2 3 4 5 2 3 4 5 2 3 4 5
6 2 2 6 2 2 6 2 2
5 2 6 1 5 15 2 11 5 15 2 15
x y z x y z x y z
x y z x y z x y z
x y z x y z x y z
' '' '' '
1 11 11 1
= = =
1 11 11 1
1 11 11 1
1 11 11 1
1 11 11 1
= = =
! !! !! !
1 11 11 1
1 11 11 1
1 11 11 1
= = =
1 11 11 1
1 11 11 1 + ++ ++ +
(1.22)
Explain the results.
Exercise 1-4: Find the eigen-values and eigen-vectors of the above matrices.
Now explain the conclusions of Equation via eigen-values and eigen-vectors.
Exercise 1-5: Consider the differential equation with boundary conditions

2
2
1; (0) (1) 0
d f
f f
dx
= = = (1.23)
Pose and solve the above differential equation via central finite difference. The
theory to convert differential equation problem into a linear algebra problem via
finite difference. Also see the diag command in Python to capture the resulting
finite difference matrix.
Exercise 1-6: Using sampling, find the minimum location (to within 0.001) of
the function
2
( ) 1
x
f x xe

= . Verify that the gradient of the function is nearly

zero at that location.
Exercise 1-7: Using sampling, find the location where the derivative of
2
( ) 1
x
f x xe

= is almost zero (to within 0.001). Compare the result with the
previous exercise.
Exercise 1-8: Write a Python function binSearch.py that asks the reader to
think of an integer between 0 & 1000. Then guess this number using no more
than 10 questions for which the user must reply with a y/n (yes or no).
Exercise 1-9: Write a Python function coins.py that asks the reader to input an
integer between 1 and 100 (cents). Then the program should return the least
number of coins to reconstruct the input integer. Allowed coins are: a penny (1
cent), nickel (5 cents), dime (10 cents) and quarter (25 cents). For examples,
suppose the user specifies 38 cents, the desired solution is 1 quarter, 1 dime, 3
pennies.
Exercise 1-10: Use graphical means to find the minimum location (to within
0.001) of the function
2
2
( ) 1
x
f x xe

= . Verify that the gradient of the function is

nearly zero at that location.
Exercise 1-11: Using contour plots, find the minimum location (to within a
radius of 0.001) of the function
( )
( )
( )
2
2 2
2
2 2
1
100 ( 1) 1
2
( , )
1
500 ( 1) 1 10 8
2
u v
f u v
u v u v
' '
1 1
1 1
1 1
1 1 1 1
=
! !
1 1
1 1

1 1
1 1
1 1 + +

Verify that the gradient of the function is nearly zero at that location.
Exercise 1-12: Modify the Polynomial class so that one can add Polynomials of
different orders. Include the following test case:

2
( ) 1 3.2
( ) 1
( ) ( ) ( ) ?
( ) ( ) ( ) ?
p x x x
q x x
r x p x q x
r x q x p x
=
=
= =
= =
(1.24)
Exercise 1-13: Expand the Polynomial class to include subtraction of
polynomials of different orders. Include the following test case:

2
( ) 1 3.2
( ) 1
( ) ( ) ( ) ?
( ) ( ) ( ) ?
p x x x
q x x
r x p x q x
r x q x p x
=
=
= =
= =
(1.25)
Exercise 1-14: Expand the Polynomial class to include multiplication of
polynomials of different orders. Include the following test case:

2
( ) 1 3.2
( ) 1
( ) ( ) ( ) ?
( ) ( ) ( ) ?
p x x x
q x x
r x p x q x
r x q x p x
=
=
= =
= =
(1.26)