Lecture Notes J2LALP
Lecture Notes J2LALP
Lecture Notes J2LALP
Lecture notes
This is the lecture notes for the module “Linear Algebra and Linear Programming”
taught for second-year students in the Jinan-Birmingham dual degree undergraduate
programmes. The notes consists of two parts. The first part (Chapters 1 & 2) is the
continuation of the module “Vectors, Geometry & Linear Algebra” taught in the first
year and develops the theory of vector spaces covering eigenvectors, characteristic poly-
nomials, inner products, and diagonalisation. The second part (Chapters 3–11) focuses
on Linear Programming (LP) covering LP modelling, geometric method, duality theory,
the simplex method and some extensions of the simplex method.
Linear algebra grew out of the development of techniques at the start of the 18th
century by Leibniz, Cramer and Gauss to solve systems of linear equations. Cayley
developed matrix algebra in the middle of the 19th century and the definition of a vector
space was made by Peano at the end of the 19th century, resulting in a theory of linear
transformations and vector spaces in the early 20th century. Linear algebra is not only
fundamental to both pure and applied mathematics, but also has applications ranging
from quantum theory to Google search algorithms.
If linear algebra grew out of the solution of systems of linear equations, then linear
programming grew out of attempts to solve systems of linear inequalities, allowing one
to optimise linear functions subject to constraints expressed as inequalities. The theory
was developed independently at the time of World War II by the Soviet mathematician
Kantorovich, for production planning, and by Dantzig, to solve complex military planning
problems. Koopmans applied it to shipping problems and the technique enjoyed rapid
development in the postwar industrial boom. The first complete algorithm to solve linear
programming problems, called the simplex method, was published by Dantzig in 1947 and
in the same year von Neumann established the theory of duality. In 1975, Kantorovich
and Koopmans shared the Nobel Prize in Economics for their work and Dantzig’s simplex
method has been voted the second most important algorithm of the 20th century after
the Monte Carlo method. Linear programming is a modern and immensely powerful
technique that has numerous applications, not only in business and economics, but also
in engineering, transportation, telecommunications, and planning.
By the end of the module students should be able to:
• understand and use the basic concepts of linear algebra and matrices, including
linear transformations, eigenvectors and the characteristic polynomial,
• understand the basic theory of inner products and apply it to questions of orthog-
onality and/or diagonalisability,
• explain the basic techniques of linear programming (graphical method and simplex
method),
• explain why and when the simplex method fails to provide a solution and how to
resolve such a situation
• present, prove and use the results of duality theory and interpret them,
1
Contents
2.3 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2
CONTENTS
3.3 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.6 LP Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6 Duality theory 95
3
CONTENTS
8.6.2 Find the dual optimal solution and objective value . . . . . . . . . 153
11.1 Dual information from the (primal) optimal simplex tableau . . . . . . . . 182
4
CONTENTS
Index 195
Bibliography 195
5
Chapter 1
In this chapter we study linear maps and their properties. The following topics will
be covered:
• rank-nullity theorem
• characteristic polynomials,
• diagonalisation.
6
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
[Tao16, pages 39-93], [Str16, Chapters 6& 8] and [KW98, Chapters 8-12] present similar
topics as in this chapter that you may find useful.
In this section we recapitulate relevant definitions and properties of linear maps that
have been covered in the module Vectors, Geometry & Linear Algebra.
Throughout this section, we denote by U and V two vector spaces over a common field
F.
Definition 1.1 (Linear map). A function (or mapping) f : U → V is called a linear map
(or linear transformation) if the following two conditions are satisfied
Note that the vector spaces U and V can be equal. Linear maps appear naturally in
many applications in economics, physics and in everyday life. Below are some examples
of linear maps.
R2 → R2
x −x
7→ ,
y −y
7
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
R2 → R2
x x
7→ ,
y −y
R2 → R2
x −x
7→ ,
y y
R2 → R2
x cos(θ)x − sin(θ)y cos(θ) − sin(θ) x
7→ = ,
y sin(θ)x + cos(θ)y sin(θ) cos(θ) y
R2 → R2
x λx
7→ .
y λy
Example 1.3 (Linear maps in economics). There are m products, P1 , . . . , Pm , that are
made up from n elements, e1 , . . . , en . Product Pi (i = 1, . . . , m) contains aij units of
element ej (j = 1, . . . , n). Let xi (i = 1, . . . , n) be the price of each unit of element ei .
What is the price list off all products?
p = Ax,
where
p a . . . a1n x
1 11 1
.. .. . ..
A = . . . . .. ,
p = . , x = . .
pm a1m . . . amn xn
8
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
Rn → Rm
x1 p1 a11 . . . a1n x
1
.. . . . .
. 7→ .. = .. . . . .. ..
xn pm a1m . . . amn xn
Example 1.4 (Linear maps in daily life: bills for the meals). We actually encounter linear
maps daily. A group of m students go for a diner. The meals are made up of n courses.
The meal for student i, i = 1, . . . , m, consists of aij units of course j, j = 1, . . . , n. The
price of a unit of course j is xj , j = 1, . . . , n. Then the price list of all students is
p a . . . a1n x
1 11 1
.. .. .. ..
. = . ... . . .
pm a1m . . . amn xn
We notice that mathematically this problem is exactly the same as in Example 1.3.
Example 1.5 (Linear maps in Physics). A typical example of a linear map is Newton’s
−
→
second law in classical mechanics expressing a linear dependence between the force F
acting on an object and the acceleration −
→
a that is generated by the force
−
→
F = m−
→
a,
where m is the mass of the object. Here U = R3 . Many other laws in Physics such as
Fourier’s law of heat conduction, Fick’s law of diffusion and Hooke’s law of elasticity also
satisfy similar linear relations.
The following proposition states basic properties of a linear map, see Theorem 10.2
in J1VGLA Lecture Notes.
(i) f (0) = 0,
9
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
To define a function f : U → V , one needs to specify what f (u) is for every single
element u ∈ U , and there are usually infinitely many such elements. The most useful
property for linear maps is that one only need to specify for a basis of the domain vector
space U (as long as U is finite dimensional). This is given in the following theorem, which
is Theorem 10.3 in J1VGLA Lecture Notes.
f (a1 u1 + . . . + an un ) = a1 v1 + . . . + an vn (1.2)
for any a1 , . . . , an ∈ F.
Proof. We first show the existence of a linear map f satisfying the condition (1.1). Let
u ∈ U . Since {u1 , . . . , un } is a basis of U , u can be written as
u = α1 u1 + . . . + αn un
f (u) := α1 v1 + . . . + αn vn .
One can verify easily that f is linear. We have u1 = 1u1 + 0u2 + . . . + 0un . Thus by
definition
f (u1 ) = 1v1 + 0v2 + . . . + 0vn = v1 .
Now suppose that there exist two linear maps f1 and f2 both satisfying (1.1). Let
u = α1 u1 + . . . + αn un ∈ U . Then by the linearity of f1 we have
10
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
We also recall the definition of coordinate of a vector with respect to a given basis.
is called the coordinates of v. The coordinate vector of v is the column vector, [v]B , defined
by
a
1
.
[v]B = .. .
an
Definition 1.9 (matrix of a linear map). Let U and V be finite-dimensional vector spaces
over a common field F with dim U = n and dim V = m. Let f : U → V be a linear map.
Let B = {u1 , . . . , un } be a basis of U and C = {v1 , . . . , vm } be a basis of V . The matrix
of f with respect to the bases B and C is the m × n matrix A with columns
Example 1.10 (matrix representation of a linear map with respect to standard bases).
Let us consider again Example 1.3 where the price list p ∈ Rm for all students was
computed from the that of the courses x by
p = Ax,
11
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
Therefore,
−2 −1
[f (u1 )]B = , [f (u2 )]B = .
−1 2
Hence, by definition the matrix representation of f with respect to B is given by
−2 −1
A= .
−1 2
The following theorem provides a relation between [f (u)]c and [u]B for a general vector
u in U via the matrix representation. This is an important result which enables the study
of a general linear map via the study of its matrix representation.
Theorem 1.12. Let U and V be finite-dimensional vector spaces over a common field F
with dim U = n, dim V = m. Let B = {u1 , . . . , un } be basis of U and C = {v1 , . . . , vm }
be a basis of V . Let A be the matrix representation of f with respect to the bases B and
C. Then for every u ∈ U , we have
[f (u)]C = A[u]B .
Proof. Let [u]B = (b1 , . . . , bn )T , so that u = j=1 bj uj . Let A be the matrix representation
Pn
12
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
Thus we obtain [f (u)]C = A[u]B as claimed and finish the proof of the theorem.
[f (u)]C = [f ]B
C [u]B .
If U = V we usually use the same basis B for U and for V . In this case, instead of saying
“the matrix of f with respect to B and B” we will say “the matrix of f with respect to
B”. The following theorem is, in some sense, a converse to Theorem 1.12.
Theorem 1.13. Let U and V be finite-dimensional vector spaces over a common field F
with dim U = n and dim V = m. Let B and C be two given ordered bases of U and V
respectively. Let A be m × n-matrix with entries in F. Then
(i) There exists a unique map f : U → V such that the matrix of f with respect to B
and C is A.
(ii) Suppose that g : U → V is a function such that for all u ∈ U we have [g(u)]C =
A[u]B . Then g is linear and g is the (unique) linear map from U to V which has
matrix representation A with respect to B and C.
1, . . . , n. By Theorem 1.7, there exists a unique linear map f : U → V such that f (uj ) =
wj for all j = 1, . . . , n. Therefore, the coordinate vector [f (uj )]C is (a1j , a2j , . . . , amj )T ,
which is the j-th column of A. Thus by definition of a matrix representation, A is the
matrix representation of f with respect to B and C.
(ii). Let f : U → V be the linear map constructed in the previous part, so that f
has matrix representation A with respect to B and C. We need to prove that f = g.
13
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
The following Theorem, cf. Theorem 10.15 and its proof in J1VGLA Lecture Notes,
provides a formula for computing the matrix representation of the composition of two
linear maps.
Theorem 1.14 (Matrix of the composition of two linear maps). Let U, V and W be
finite-dimensional vector spaces over a field F with bases B, C and D respectively. Let
f : U → V and g : V → W be linear maps. Let K be the matrix of f with respect to B
and C and let L be the matrix of g with respect to C and D. Then LK is the matrix of
the composition of g and f , g ◦ f , with respect to B and D.
Definition 1.15 (Kernel, image, rank and nullity of a linear map). Let f : U → V be a
linear map. The kernel (also known as the null space), ker(f ), the image (also known as
the range), im(f ), the , rank(f ) and the nullity, nullity(f ), of f are defined respectively
by
The following lemma shows that ker(f ) and im(f ) are two subspaces of U and V respec-
tively.
14
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
𝑈
𝑓: 𝑈 → 𝑉 𝑉
𝐾𝑒𝑟(𝑓)
𝐼𝑚(𝑓)
0
0
Figure 1.1: Kernel and image of a linear map f from a vector space U to a vector space
V . The kernel of f which contains all pre-image of the vector 0 is a subspace of U while
the image of f , which consists of all f (u), with u ∈ U , is a subspace of V .
The following theorem states an important relation between the rank, the nullity and
the dimension of U .
15
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
the whole of im(f ). Let v ∈ im(f ). Then v = f (u) for some u ∈ U . Since {u1 , . . . , un } is
a basis of U , we have u = a1 u1 + . . . + an un for some a1 , . . . , an ∈ F. Using the linearity
of f , we have
where we have used the fact that f (u1 ) = . . . = f (uk ) = 0 since u1 , . . . , uk ∈ ker(f ). The
above equation shows that v ∈ span(f (uk+1 ), . . . , f (un )), thus im(f ) ⊂ span(f (uk+1 ), . . . , f (un )).
Therefore im(f ) = span(f (uk+1 ), . . . , f (un )). We now prove that the set B is linearly in-
dependent. Suppose that there exist b1 , . . . , bn−k ∈ F such that
b1 uk+1 + . . . + bn−k un = c1 u1 + . . . + ck uk
c1 u1 + . . . + ck uk − b1 uk+1 − . . . − bn−k un = 0.
Since {u1 , . . . , un } is a basis of U (thus they are linearly independent), we deduce that
c1 = . . . = ck = b1 = . . . = bn−k = 0.
Hence B is linearly independent and thus, it is a basis of ker(f ), and we obtain that
16
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
[u1 ]B 0 , . . . , [un ]B 0 .
Example 1.21 (example of transition matrix). Let U = R2 and let B = {u1 , u2 } and
B 0 = {u01 , u02 } be two ordered bases U where
1 1 2 3
u1 = , u2 = , u01 = , u02 = .
0 1 1 2
We have
1 2 3 2
u1 = = 2 − 1 = 2u01 − u02 ; thus [u1 ]B 0 = ,
0 1 2 −1
1 2 3 −1
u2 = = − + = −u01 + u2 ; thus [u2 ]B 0 = .
1 1 2 1
17
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
The transition matrix is useful when one want to change coordinate systems.
[v]B 0 = P [v]B .
v = α1 (p11 u01 + p21 u02 + . . . + pn1 u0n ) + . . . + αn (p1n u01 + p2n u02 + . . . + pnn u0n )
18
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
Example 1.23 (change of coordinate system). The change of coordinate systems plays an
important role in many real applications such as computer graphics and global positioning
system (GPS). Example let U = R2 , let B = {e1 , e2 } be the standard basis (coordinate
system) of U and let B 0 = {v1 , v2 } be another coordinate system where
√ √
1 3 1 3
v e + 2 e2 e
1 = 2 √1 = 2√ 2 1
.
v2 − 23 e1 + 12 e2 − 23 12 e2
Theorem 1.24. Let B and B 0 be two ordered bases of U . Let P be the transition matrix
from B to B 0 . Then P is invertible and P −1 be the transition matrix from B 0 to B.
19
1.1. LINEAR MAPS, MATRIX REPRESENTATION, RANK-NULLITY THEOREM
Now we will use this claim to prove the statement of the Theorem. Let Q be the
transition matrix from B 0 to B. Let v ∈ U be an arbitrary vector in U . By applying
Theorem 1.22 twice, the first time for P and the second one for Q, we have
where the last equality is due to the associativity of matrix multiplication. Since every
element of Fn can be written in the form [v]B 0 for some v ∈ U , the above equation implies
that
w = (P Q)w for all w ∈ Fn ,
or equivalently
(P Q − I)w = 0 for all w ∈ Fn .
Remark 2. Because of Theorem 1.24, some textbooks/sources will define the transition
matrix as being P −1 .
To conclude this section, we include two known theorems that will be useful later on.
20
1.2. CHANGE OF BASIS AND SIMILAR MATRICES
Proof. See Chapter 8 in the lecture notes J1VGA of Vectors, Geometry & Linear Algebra.
Theorem 1.26. Let A be an n × n matrix over a field F. Then the following statements
are equivalent
(i) det(A) = 0,
Proof. See Corollary 8.34 and and Corollary 8.35 in the lecture notes J1VGA of Vectors,
Geometry & Linear Algebra.
Theorem 1.27 (change of basis). Let B = {u1 , . . . , un } and B 0 = {u01 , . . . , u0n } be two
ordered bases of U . Let P be the transition matrix from B to B 0 . Let A and A0 be the
matrix of f with respect to B and B 0 respectively. Then
A0 = P AP −1 .
21
1.3. DIAGONAL AND DIAGONALISABLE MATRICES
By Theorem 1.24, P −1 be the transition matrix from B 0 to B. Using Theorem 1.22 again
we get
[u]B = P −1 [u]B 0 . (1.8)
Inspired by the above theorem, we consider the following definition of similarity be-
tween two matrices.
Definition 1.28 (similar matrices). Two n × n-matrices A and B are said to be similar
if there is an invertible matrix P such that B = P AP −1 .
1 2 −6 −4
Example 1.29. A = is similar to B = since B = P AP −1 , which
3 4 16 11
can be verified by direct computations, where
1 −1
P = .
−1 2
In subsequent sections, we will see that similar matrices have many interesting prop-
erties in common.
Matrices that arise from real applications are often of high dimension. Computations
with such matrices are complicated and costly. For instance, suppose that one wants to
raise a matrix A to a large power, say A1001 . Using usual matrix multiplication will be
22
1.3. DIAGONAL AND DIAGONALISABLE MATRICES
extremely hard. In such situations, one wants to simplify complex matrices into simpler
ones. One particular class of matrices that particularly facilitate such tasks are diagonal
matrices.
• matrix multiplication:
In particular,
diag(a1 , . . . , an )2 = diag(a21 , . . . , a2n ),
and more generally, raising diagonal matrices to high power can be done easily: for any
k∈N
diag(a1 , . . . , an )k = diag(ak1 , . . . , akn ).
f (A) = 5A3 − 4A2 + 3A + 2I = diag(5a31 − 4a21 + 3a1 + 2, . . . , 5a3n − 4a2n + 3an + 2).
23
1.4. EIGENVALUES AND EIGENVECTORS
−1 2
Example 1.33. A = is diagonalisable since A is similar to the diagonal
−1 2
1 0 1 1
matrix D = . This is because A = P DP −1 where P = .
0 3 1 −1
Computations with diagonalisable matrices are also easy: for any polynomial f and
a diagonalisable matrix A with A = P −1 DP where D = diag(d1 , . . . , dn ) then
Uλ := {v ∈ U : f (v) = λv}
Above idU is the identity matrix on U , that is idU : U → U satisfying idU (u) = u for
all u ∈ U .
24
1.4. EIGENVALUES AND EIGENVECTORS
Example 1.39.
• A = diag(a1 , . . . , an ) and f (v) = Av, then the basis vectors e1 , . . . , en are eigenvec-
tors for f with eigenvalues a1 , . . . , an respectively.
1 4
• f : R2 → R2 with f (u) = Au for all u ∈ R2 where A = . Then (2, 1)T is an
1 1
eigenvector of f with eigenvalue 3 since
2 1 4 2 6 2
f = = = 3 .
1 1 1 1 3 1
The following theorem shows the equivalence between the set of eignevalues of a linear
transformation with that of its matrix representation with respect to some basis.
25
1.4. EIGENVALUES AND EIGENVECTORS
Because of the above theorem, it is sometimes more convenient when using matrix
representation.
Lemma 1.41. Suppose that U is an n-dimensional vector space and suppose that f :
U → U is a linear transformation. Suppose that U has an ordered basis B = (u1 , . . . , un )
such that each uj is an eigenvector of f with eigenvalue λj . Then the matrix A of f with
respect to B is diagonal: A = diag(λ1 , . . . , λn ). Conversely, if B = (u1 , . . . , un ) is a basis
such that the matrix A of f with respect to B is diagonal, A = diag(λ1 , . . . , λn ) then each
uj , j = 1, . . . , n is an eigenvector of f with eigenvalue λj .
Q = (v1 , . . . , vn ).
We will provide two proofs using different knowledge studied previously. The first
proof will use Theorem 1.27 on the formula for a change of basis. In this proof, one
combines matrix representation, transition matrix, with the recently introduced concepts
of eigenvectors and eigenvalues. The second proof directly applies matrix multiplications.
First proof. We will use Theorem 1.27. Let B 0 = (v1 , . . . , vn ) be the ordered basis of Rn
given in the hypothesis of the lemma. Let B = (e1 , . . . , en ) be the standard basis of Rn .
26
1.4. EIGENVALUES AND EIGENVECTORS
f (vj ) = Avj = λj vj ,
A = Qdiag(λ1 , . . . , λn )Q−1
as claimed.
= Qdiag(λ1 , . . . , λn ),
which means that AQ = Qdiag(λ1 , . . . , λn ). The above computations follow just from
the definition of matrix multiplications (remembering that v1 , . . . , vn are column vectors).
Since {v1 , . . . , vn } forms a basis of Rn , they are linearly independent. Thus Q is invertible.
Therefore, we get A = Qdiag(λ1 , . . . , λn )Q−1 as claimed
α1 v1 + . . . + αm vm = 0. (1.9)
27
1.4. EIGENVALUES AND EIGENVECTORS
By applying both sides of this equation with (A − λm−1 I) and using again the argument
above we eliminate vm−1 and get
By the hypothesis {λi } are all distinct, thus (λ1 − λm )(λ1 − λm−1 ) . . . (λ1 − λ2 ) is non-zero.
Since α1 is also non-zero, we must have v1 = 0. This contradicts with the fact that v1 is
an eigenvector. Hence the assumption that v1 , . . . , vm are linearly dependent can not be
true, that is they are linearly independent.
Corollary 1.44. Let λ1 , . . . , λm ∈ F be distinct and let Uλ1 , . . . , Uλm be the corresponding
eigenspaces. If Cj is a basis of Uλj for each j = 1, . . . , m, then the set
C1 ∪ C2 ∪ . . . ∪ C m
is linearly independent.
C1 ∪ C2 ∪ . . . ∪ Cm = {wi,j : 1 ≤ j ≤ m, 1 ≤ i ≤ dj }.
Suppose that
dj
m X
(1.11)
X
aij wij = 0
j=1 i=1
28
1.5. CHARACTERISTIC POLYNOMIAL
for some scalars αij ∈ F. We need to show that all aij are zero. For each j = 1, . . . , m,
define
dj
X
vj = aij wij .
i=1
Since Cj is basis of Uλj and Uλj is a subspace of U , we have vj ∈ Uλj . Equation (1.11)
becomes
m
X
vj = 0.
j=1
If at least one of v1 , . . . , vm is non-zero then the equality above means that we have a
linearly dependent set of eigenvectors with distinct eigenvalues which is contradicting
Theorem 1.43. Hence vj = 0 for all j = 1, . . . , m. By definition of vj , we obtain
dj
X
aij wij = 0.
i=1
Lemma 1.41 and Corollary 1.44 provide a strategy for diagonalising a linear transfor-
mation f : first find all the eigenvalues of λ1 , . . . , λm of f , then find a basis Cj for each
eigenspace Uλj and consider C = C1 ∪ C2 ∪ . . . ∪ Cm . Then by Corollary 1.44 C is linearly
independent and if the size of C is n then C is a basis of U , with respect to which f is
represented bi a diagonal matrix according to Lemma 1.41.
29
1.5. CHARACTERISTIC POLYNOMIAL
Remark 3. Note that, since det(λI −A) = (−1)n det(A−λI), det(λI −A) and det(A−λI)
have the same roots. Because of this reason, the characteristic polynomial of A can be
alternatively defined as det(A − λI) as in some literature.
The characteristic of A is
λ − 7 −10
χA (λ) = det = (λ − 7)(λ + 8) + 50 = λ2 + λ − 6 = (λ − 2)(λ + 3).
5 λ+8
This system is equivalent to x = −2y. Thus Uλ1 = {(x, y)T ∈ R2 : x = −2y} which has a
basis (−2, 1)T . Similarly to find eigenvectors corresponding to λ2 , we solve
7 10 x x
= −3 ,
−5 −8 y y
which is equivalent to
Solving this system gives x = −y. Thus Uλ2 = {(x, y)T : x = −y} which has a basis
(1, −1)T .
30
1.5. CHARACTERISTIC POLYNOMIAL
The characteristic of A is
λ − 2 −3
χA (λ) = det = (λ − 2)(λ + 2) + 9 = λ2 + 5.
3 λ+2
Proof. Suppose that A and B are similar matrices, so that B = P −1 AP for some invertible
matrix P . We recall the following property of a determinant: for square matrices of equal
size A1 , A2 :
det(A1 A2 ) = det(A1 ) det(A2 ),
= χA (λ).
i=1
Proof. This can be proved by induction. See [Tao16, Theorem 4, page 179]
Example 1.52.
31
1.5. CHARACTERISTIC POLYNOMIAL
a11 a12
• For n = 2, A= , then
a21 a22
a11 − λ a12
χA (λ) = det = (a11 − λ)(a22 − λ) − a12 a21
a21 a22 − λ
= λ2 − tr(A)λ + det(A).
a 0 0
• For n = 3, A = 0 b 0, then
0 0 c
λ−a 0 0
χA (λ) = det 0 λ−b 0 = (λ − a)(λ − b)(λ − c).
0 0 λ−c
From now on we assume that F = C (unless otherwise specified). This is due to the
following important theorem.
Theorem 1.53 (Fundamental theorem of algebra). Every polynomial with real or complex
coefficients has at least one complex root.
Corollary 1.54. Let p = p(z) be a non-zero polynomial of degree n with complex coeffi-
cients. Then p(z) can be written in the form
The fundamental theorem of algebra is one of the most important theorems in Math-
ematics and has a long history. Many proofs have been proposed including those by
d’Alembert in 1746, Euler (1749), de Foncenex (1759), Lagrange (1772), and Laplace
32
1.5. CHARACTERISTIC POLYNOMIAL
(1795). A rigorous proof was first published by Argand in 1806 and revised in 1813.
For more information on the history of the fundamental theorem of algebra, one can see
[Sma81]. As far as I know (I do not have Argand’s paper), the following well-known proof
using the extreme value theorem, that is a real-valued continuous function on a bounded
interval must attain a minimum, is very close to the one by Argand.
p(z) = p0 + pm z m + . . . + pn z n ,
where pm 6= 0 is the first nonzero coefficient of p(z) following p0 . We will show that there
exists a point z ∗ such that |p(z)| < |p0 | which would contradict the assumption that p
attains its global minimum at 0. Define
∗ p0 1/m
z =r −
pm
where r is a small positive number which will be chosen later, and (−p0 /pm )1/m denotes
any of the m-roots of −p0 /pm . Clearly z ∗ 6= 0. We evaluate the value of p at z ∗ :
33
1.5. CHARACTERISTIC POLYNOMIAL
Let s := r(|p0 /pm |)1/m . Using the triangle inequality, we can estimate the remainder as
follows
n
X
|R| ≤ ri pi (p0 /pm )i/m
i=m+1
p0 (m+1)/m
≤ rm+1 max |pi |(1 + s + . . . + sn−m−1 )
pm m+1≤i≤n
p0 (m+1)/m 1 − sn−m
= rm+1 max |pi |
m+1≤i≤n pm 1−s
r m+1
maxm+1≤i≤n |pi | p0 (m+1)/m
≤ .
1−s pm
Since the right-hand side tends to 0 as r tends to 0, for sufficiently small enough r, we
have |R| < |p0 |rm /2. Thus for such an r, we have
|p(z ∗ )| = |p0 − p0 rm + R| ≤ |p0 − p0 rm /2| = |p0 |(1 − rm /2) < |p0 | = |p(0)|.
This is a contradiction with the assumption that |p| attains its global minimum at the
origin.
The number mj in the theorem above is called the multiplicity of the root λj (j =
1, . . . , k). We now apply Theorem 1.53 to the characteristic polynomial χA of an n × n
matrix A
χA (λ) = (λ − λ1 )m1 (λ − λ2 )m2 . . . (λ − λk )mk , (1.12)
di ≤ mi ∀i = 1, . . . , k.
34
1.5. CHARACTERISTIC POLYNOMIAL
where ∗ indicates an unknown entry. Applying Theorem 1.25 part (iv), we obtain that
for some polynomial p(t) which is the characteristic of the bottom right block in A0 . By
Theorem 1.27, A and A0 are similar and hence by Theorem 1.49 we have χA (λ) = χA0 (λ) =
(λ − λi )di p(t). According to Theorem 1.53 the biggest power of λ − λi dividing χA (λ) is
(λ − λi )mi . This implies that di ≤ mi as claimed.
35
1.6. AN ALGORITHM FOR DIAGONALISATION
D = diag(λ1 , . . . , λ1 , . . . , λk , . . . , λk )
with distinct λ1 , . . . , λk ∈ C.
Step 2. For each j = 1, . . . , k find a basis vj,1 , . . . , vj,dj for the eigenspace Uλj with
dj = dim(Uλj ).
• If dj = mj for all j then A is diagonalisable and P is the matrix whose columns are
the basis vectors and D is the diagonal matrix D = diag(λ1 , . . . , λ1 , . . . , λk , . . . , λk )
where each λj appear dj times.
36
1.6. AN ALGORITHM FOR DIAGONALISATION
So A has only one the eigenvalue λ1 = 1 with algebraic multiplicity m1 = 2. Let us find
eigenvectors with respect to λ1 . We need to solve
1 1 x x
= ,
0 1 y y
x + y = x, y = y.
so A has two distinct eigenvalues λ1 = 2 and λ2 = −3. Moreover, Uλ1 has a basis
v1 = (−2, 1)T and Uλ2 has a basis v2 = (1, −1)T . Since m1 = m2 = 1, f is diagonalisable.
The matrix of f with respect to the basis
−2 1
B0 =
1 −1
37
1.6. AN ALGORITHM FOR DIAGONALISATION
Example 1.60. Let f : C3 → C3 be the standard linear map associated with the matrix
1 2 −4
A = −10 −11 20 .
−6 −6 11
that is
2x1 + 2x2 − 4x3 = 0
−10x1 − 10x2 + 20x3 = 0
−6x1 − 6x2 + 12x3 = 0.
38
1.6. AN ALGORITHM FOR DIAGONALISATION
B 0 = {v1 , v2 , v3 }
Example 1.61.
1 1 0
A = 0 1 1
0 0 1
Similarly as in Example 1.58, A is non-diagonalisable since
Example 1.62.
−26 3
A = 0 −2 0
4 0 −3
The characteristic polynomial of A is
λ−6 −3 2
det(λI − A) = det 0
λ+2 0
−4 0 λ+3
λ−6 2
= (λ + 2) det
−4 λ + 3
h i
= (λ + 2) (λ − 6)(λ + 3) + 8
= (λ + 2)2 (λ − 5).
39
1.7. SOME APPLICATIONS OF DIAGONALISATION
Thus
λ1 = −2, m1 = 2; λ2 = 5, m1 = d1 = 1.
is called a system of difference equations. These type of equations appear in many practical
applications where the vector xk is often used to model the “state” of a biological/phys-
ical/economical system at time (day/month/year) k given the initial state x0 . Suppose
that A is diagonalisable. One can solve (1.14) as follows. We have
x1 = Ax0
x2 = Ax1 = A(Ax0 ) = A2 x0
x3 = Ax2 = A(A2 x0 ) = A3 x0
..
.
40
1.7. SOME APPLICATIONS OF DIAGONALISATION
Ak = P Dk P −1 thus xk = P Dk P −1 x0 ,
The Fibonacci sequence: The sequence of Fibonacci numbers Fn are defined re-
cursively as follows: given F0 = 0, F1 = 1 and for n ≥ 2
Proof. There are many possible ways to prove this theorem. Here we will show a proof
using Linear Algebra. For k ≥ 1, we define
Fk
uk = .
Fk−1
1
Thus u1 = and from (1.15) we have
0
Fk+1
Fk + Fk−1 1 1 Fk
uk+1 = = = = Auk ,
Fk Fk 1 0 Fk−1
1 1
where A = . It follows that for all n ≥ 2,
1 0
41
1.7. SOME APPLICATIONS OF DIAGONALISATION
that is
F 1
n+1 = An . (1.17)
Fn 0
The key now is to calculate An which is n-th power of matrix A. This will be efficiently
done via diagonalisation. First we need to find eigenvalues of A. The characteristic
polynomial of A is
λ − 1 −1
pA (λ) = det(λI − A) = det = λ2 − λ − 1.
−1 λ
A = P DP −1 ,
where
λ1 λ2 1 1 −λ2 λ1 0
P = , P −1 = , D = .
1 1 λ1 − λ2 −1 λ1 0 λ2
Thus
1 λ1 λ2 λn1 0 1 −λ2
An = P Dn P −1 =
λ1 − λ2 1 1 0 λn2 −1 λ1
Substituting this formula into (1.17) we get
n n+1 n+1
Fn+1 1 λ λ λ 0 1 −λ 1 1 λ − λ
= 1 2 1 2
= 1 2
.
Fn λ 1 − λ2 1 1 0 λ2 n
−1 λ1 0 λ 1 − λ 2 n
λ1 − λ2n
42
1.7. SOME APPLICATIONS OF DIAGONALISATION
One can derive many interesting properties of Fibonacci numbers using explicit for-
mula obtained in Theorem 1.63. The next proposition states a connection between the
so-called golden ration with Fibonacci numbers.
The ratio above is known as the golden ratio which appears ubiquitously in architec-
ture and designs.
where to obtain the last equality we have use the fact that
1 − √ 5 n
lim √ = 0,
n→∞ 1 + 5
√
which is because 1−√5
1+ 5
< 1. This finishes the proof of this proposition.
Using the method in the proof of Theorem 1.63 and Proposition 1.64 one can find
explicit formula and properties for similar sequences that are defined recursively.
43
1.7. SOME APPLICATIONS OF DIAGONALISATION
We knew that the solution to the following ordinary differential equation (ODE)
dx(t)
= ax(t), x(0) = x0 ,
dt
x(t) = eat x0 .
dx(t)
= Ax(t), x(0) = x0 , (1.18)
dt
where x0 ∈ Rn and A ∈ Rn×n are given. How do one solve this system? Can one define
the exponential of a matrix so that
x(t) = eAt x0 ?
We recall that the exponential function can be defined by the following power series
∞
x
X xi x 2 x3
e := =1+x+ + + ....
i=0
i! 2 6
In the above formula, the right hand sides consist of powers of x. We have known how
to define powers of a matrix. Inspired by this, we can define exponential of a matrix as
follows.
When A is diagonalisable one can find exp(A) using the formula exp(A) = P −1 exp(D)P
with exp(D) = diag(ed1 , . . . , edn ). Once the exponential function of a matrix has been
defined, one can solve the system (1.18): its solution can be given as x(t) = exp(At)x0 .
y 0 (t) = x(t),
y(0) = y0 .
44
1.7. SOME APPLICATIONS OF DIAGONALISATION
Let
0 1 x(t) x0
A= , x(t) := , x(0) =
1 0 y(t) y0
the above system can be written in a matrix form as in (1.18):
dx(t)
= Ax(t), x(0) = x0 . (1.19)
dt
45
1.8. JORDAN CANONICAL FORM
As we have seen in the previous section and examples, not every matrix is diagonalis-
able. In this section we introduce a more general form than diagonal, namely the Jordan
canonical form, that every matrix can be represented in this form.
Definition 1.67 (Jordan block). Let λ ∈ C and ` ∈ N. The Jordan λ-block is of size ` is
the ` × `-matrix
λ 1 0 0 ... 0
0 λ 1 0
. . . 0
0 0 λ 1 . . . 0
..
J` (λ) = .. . . . . . . .. .
. . . . . .
0 0 ... 0 λ 1
0 0 ... 0 0 λ
Thus J` (λ) is a bidiagonal matrix where the entries are equal to λ on the main diagonal
and to 1 on the diagonal above. For instance
λ 1 0
λ 1
J1 (λ) = (λ), J2 (λ) = , J3 (λ) = 0 λ 1 .
0 λ
0 0 λ
A proof of Theorem 1.68 is technical and is out of the scope of this module. For
further information, see for instance [Str06, Appendix B]. The Jordan canonical form
46
1.8. JORDAN CANONICAL FORM
plays an important role in solving linear equations because of two reasons (1) it has a
very simple structure (upper triangular, and only 1s just above the diagonal) and (2) it
can be computed for any square matrix by Theorem 1.68. For instance for the matrix A
in Example 1.62 (recalling that A is non-diagonalisable)
6 3 −2
A = 0 −2 0
4 0 −3
47
Chapter 2
In previous chapters, we have studied vector spaces and various important concepts
that are concerned with the two basis operations equipped to a vector space (addition and
scalar multiplications) such as bases, linear transformations, eigenvalues and eigenvectors
and diagonalisation. In this chapter, we will study another important operation with
vectors, namely inner products. Via inner products, we will be able to study geometrical
properties of vectors such as orthogonality and length of a vector. The following topics
will be covered:
• bilinear forms,
• orthogonality,
• inner products,
• length of a vector,
• Gram-Schmidt orthogonalisation.
48
2.1. SYMMETRIC BILINEAR FORMS
[Tao16, pages 175-221], [Str16, Chapter 4] and [KW98, Chapters 3-5] contain relevant
material to this chapter that you may find helpful.
(B1) Φ(u + v, w) = Φ(u, w) + Φ(v, w) and Φ(au, v) = aΦ(u, v) for all u, v, w ∈ U and
a ∈ F.
(B2) Φ(u, v + w) = Φ(u, v) + Φ(u, w) and Φ(u, av) = aΦ(u, v) for all u, v, w ∈ U and
a ∈ F.
49
2.1. SYMMETRIC BILINEAR FORMS
If the identity Φ(u, v) = Φ(v, u) is satisfied for all u, v ∈ U then in order to prove
bilinearity of Φ it suffices to check either (B1) or (B2), the other condition follows auto-
matically.
For notational convenience, we often refer to a bilinear form as (−, −) and write (u, v)
when the arguments are explicit.
It is easy to verify that the standard scalar product is a symmetric bilinear form.
Lemma 2.5. The form on Fn associated with the matrix G defined in (2.3) is a bilinear
form.
50
2.2. THE GRAM MATRIX OF A BILINEAR FORM
where to obtain the third equality we have switched the roles of indices i and j. Hence if
G is symmetric, so is the bilinear form (·, ·)G .
then
n X
X n n
X
(u, v)G = gij xi yj = xi y i .
i=1 j=1 i=1
x1 y1
(x, y)G = , = x1 y1 + 2x1 y2 + 2x2 y1 + x2 y2 .
x2 y2
G
Definition 2.7 (The Gram matrix). . The Gram matrix of the bilinear form (·, ·) with
respect to the basis B is the n × n-matrix G = (gij )1≤i,j≤n over F that is defined by
gij = (ui , uj ).
The Gram matrix is useful since we can calculate the values of the form at any pair of
arguments using their coordinates. The following theorem transforms a general bilinear
form on U into a bilinear form on Fn that is associated to the Gram matrix G.
Theorem 2.8. Let G be the Gram matrix of the bilinear form (·, ·) with respect to a given
basis B of U . Then for any u, v ∈ U we have
51
2.2. THE GRAM MATRIX OF A BILINEAR FORM
Proposition 2.9. The Gram matrix of a symmetric bilinear form with respect to any
basis of U is symmetric.
Example 2.10. Let U = P2 (R) be the vector space of polynomials of degree at most 2
together with the zero polynomial with real coefficients. We consider a form on U defined
by Z 1
(u, v) = u(x)v(x) dx. (2.5)
−1
By linearity of integrals, it is easy to check that the above form defines a symmetric bilinear
form on U . In fact, we verify (2.1), for u, v, w ∈ U
Z 1 Z 1 Z 1
(au + v, w) = (au(x) + v(x))w(x) dx = a u(x)w(x) dx + v(x)w(x) dx
−1 −1 −1
52
2.3. ORTHOGONALITY
Thus the bilinear form is symmetric. We now compute the Gram matrix of this bilinear
form with respect to the basis B = {1, x, x2 } of U . We have
Z 1
g11 = (1, 1) = 1 dx = 2,
−1
Z 1
x2 1
g12 = (1, x) = x dx = = 0,
−1 2 −1
Z 1
2 x3 1 2
g13 = (1, x ) = x2 dx = = ,
−1 3 −1 3
Z 1
2
g22 = (x, x) = x2 dx = ,
−1 3
Z 1
x4 1
g23 = (x, x2 ) = x3 dx = = 0,
−1 4 −1
Z 1
2 2 x5 1 2
g33 = (x , x ) = x4 dx = = .
−1 5 −1 5
By symmetry we get
2
g21 = g12 = 0, g31 = g13 = and g32 = g23 = 0.
3
Thus we obtain
2
2 0 3
2
G = 0 3 0 .
2 2
3
0 5
2.3 Orthogonality
Definition 2.11. We say that vectors u, v ∈ U are orthogonal with respect to (·, ·), and
write u ⊥ v, if (u, v) = 0.
53
2.4. INNER PRODUCTS AND INNER PRODUCT SPACES
since
(u, v)G = uT Gv = 0.
However they are not orthogonal with respect to the standard scalar product (·, ·) on R2
since
(u, v) = −2 6= 0.
Theorem 2.12. Let {u1 , . . . , un } be orthogonal vectors in U such that (ui , ui ) 6= 0 for all
i = 1, . . . , n. Then u1 , . . . , un are linearly independent.
For each i = 1, . . . , n, multiplying the above equality with ui and using the fact that
(ui , uj ) = 0 for all i 6= j, we get
n
X n
X
0=( αj uj , ui ) = αj (uj , ui ) = αi (ui , ui ).
j=1 j=1
Definition 2.13 (Inner products and inner product spaces). Let (·, ·) be a symmetric
bilinear form on U . We say that (·, ·) is an inner product if it is positive definite, that is
the two following conditions are satisfied
The vector space U equipped with an inner product is called an inner product space.
54
2.4. INNER PRODUCTS AND INNER PRODUCT SPACES
Example 2.14. Let U = Rn and (·, ·) be the standard scalar product, i.e.,
n
∀u, v ∈ Rn .
X
T
(u, v) = u v = ui vi
i=1
and (u, u) = 0 implies that ui = 0 for all i = 1, . . . , n that is u = 0. Hence the standard
scalar product is an inner product and Rn equipped with this inner product is an inner
product space.
and moreover, the equality holds if and only if f (x) = 0 for all x ∈ [a, b].
Example 2.16. We recall the bilinear form in Example 2.10: U = P2 (R) with the sym-
metric bilinear form defined in (2.5). Then for u ∈ U we have
Z 1
(u, u) = u(x)2 dx ≥ 0,
−1
and moreover (u, u) = 0 implies that u = 0. Thus the symmetric bilinear form (2.5) is an
inner product and P2 (R) equipped with it becomes an inner product space.
Example 2.17. Let (·, ·) be the bilinear form on R4 associated with the matrix
1 0 0 0
0 1 0 0
G = diag(1, 1, 1, −1) =
.
0 0 1 0
0 0 0 −1
55
2.5. LENGTH OF A VECTOR, PYTHAGORAS’S THEOREM AND
CAUCHY-SCHWARTZ INEQUALITY
Since G is symmetric, the bilinear form is also symmetric. However, it is not positive
definite since, for example with u = (0, 0, 0, 1)T then (u, u) = −1 < 0. This form plays
an important role in Special Relativity, in the context of 4-dimensional space time, with
−1 corresponding to the time coordinate and the 1s corresponding to the three spatial
coordinates.
Theorem 2.18. Let (·, ·) be a symmetric bilinear form on U . Suppose that U has an
orthogonal basis {u1 , . . . , un } with (ui , ui ) > 0 for all i = 1, . . . , n. Then (·, ·) is positive
definite, that is it is an inner product on U .
For i = 1, . . . , n, we define ai = (ui , ui ). Then ai > 0. Using the fact that (ui , uj ) = 0 for
j 6= i, we have
n
X n
X
(ui , u) = (ui , xj uj ) = xj (ui , uj ) = xi (ui , ui ) = ai xi .
j=1 j=1
Now we compute
n
X n
X n
X
(u, u) = ( xi ui , u) = xi (ui , u) = ai x2i ≥ 0.
i=1 i=1 i=1
In inner products spaces, we are able to introduce geometrical concepts between vec-
tors such as length of a vector, distance and angle between two vectors. We can also
generalise a well-known theorem, namely Pythagoras’s theorem for right triangles, to a
more general context.
Let U be an inner product space equipped with inner product (·, ·).
56
2.5. LENGTH OF A VECTOR, PYTHAGORAS’S THEOREM AND
CAUCHY-SCHWARTZ INEQUALITY
Example 2.20. Let U = Rn be equipped with the standard scalar product. Then the length
of a vector u = (u1 , . . . , un )T ∈ Rn is given by the classical formula
p q
kuk = (u, u) = u21 + . . . + u2n .
Example 2.21. For the inner product (2.5) on P2 (R) in Example 2.10, the length of the
vector 1 + x + x2 is
sZ
1
k1 + x + x2 k = (1 + x + x2 )2 dx
−1
sZ
1
= (1 + 2x + 3x2 + 2x3 + x4 ) dx
−1
r r
x 4 x5 1 22
= (x + x2 + x3 + + )|−1 = .
2 5 5
as claimed.
Proof. If v = 0 then the inequality holds true since both sides are equal to 0. We consider
the case v 6= 0. We define
(u, v)
λ= .
kvk2
57
2.5. LENGTH OF A VECTOR, PYTHAGORAS’S THEOREM AND
CAUCHY-SCHWARTZ INEQUALITY
Then we have
0 ≤ ku − λvk2
which implies that (u, v)2 ≤ kuk2 kvk2 , that is |(u, v)| ≤ kukkvk. Moreover, if the equality
holds, then u = λv, that is u and v is linearly dependent. Conversely, if u and v are
linearly dependent then there exists a non-zero a such that u = av. Then we have
In other words, the equality holds if and only if u and v are linearly dependent.
Example 2.25. Let U = C([a, b], R) be the set of all continuous functions from [a, b] to
R. Then for f, g ∈ U we can define an inner product by
Z b
(f, g) := f (x)g(x) dx.
a
Definition 2.26 (distance between two vectors). Let (·, ·) be an inner product on a real
vector space V . The distance d(u, v) between two vectors u and v in V is defined by
d(u, v) = ku − vk.
58
2.6. GRAM-SCHMIDT ORTHOGONALISATION
Definition 2.27 (angle between two vectors). Let (·, ·) be an inner product on a real
vector space V . The angle between non-zero vectors u and v in V is the unique element
θ ∈ [0, π] such that
(u, v) = kukkvk cos θ.
|(u, v)|
0≤ ≤ 1,
kukkvk
Suppose that (·, ·) is a symmetric bilinear form on a vector space U over R. Let
u1 , . . . , un be vectors in U . From these vectors, we wish to construct a family of or-
thogonal vectors v1 , . . . , vn that hv1 , ..., vn i = hu1 , . . . , un i. The following Gram-Schmidt
orthogonalisation process allows us to do so
Gram-Schmidt process
v1 = u1 ,
(u2 , v1 )
v2 = u2 − v1 ,
(v1 , v1 )
(u3 , v1 ) (u3 , v2 )
v3 = u3 − v1 − v2 ,
(v1 , v1 ) (v2 , v2 )
..
.
k−1
X (uk , vi )
vk = uk − vi ,
i=1
(vi , vi )
..
.
n−1
X (un , vi )
vn = un − vi .
i=1
(vi , vi )
Note that in the computations, if (vk , vk ) = 0 for some 1 ≤ k ≤ n − 1 then we can not
compute vk+1 since that would require dividing by 0. In this case, the process stops at
the k-th step. If (vk , vk ) 6= 0 for all k ∈ {1, . . . , n − 1}, then we obtain n new vectors
59
2.6. GRAM-SCHMIDT ORTHOGONALISATION
Theorem 2.29. Suppose that (·, ·) is an inner product on U and that {u1 , . . . , un } is a
basis of U . Then the Gram-Schmidt constructed above gets to the end and {v1 , . . . , vn }
form an orthogonal basis of U .
Proof. We first note that if vj is any vector constructed by the Gram-Schmidt process,
then by the formula defining {vk }1≤k≤j , vj belongs to the span of {u1 , . . . , uj }, i.e., vj ∈
hu1 , . . . , uj i. Hence if vk is constructed during the process, then
since vj ∈ hu1 , . . . , uk−1 i for all j = 1, . . . , k − 1. Suppose that the Gram-Schmidt does
not get to the end, that is (vk , vk ) = 0 for some 1 ≤ k ≤ n − 1. Then by definition of the
inner product vk = 0. This implies that
k−1
X (uk , vi )
uk = vi .
i=1
kvi k2
60
2.6. GRAM-SCHMIDT ORTHOGONALISATION
So uk ∈ hv1 , . . . , vk−1 i. Together with (2.6) this implies that uk ∈ hu1 , . . . , uk−1 i. This
means that u1 , . . . , uk are linearly dependent which is a contradiction since {u1 , . . . , un } is
a basis of U . Therefore vk 6= 0 for all k = 1, . . . , n − 1. So the Gram-Schmidt does get to
the end. In addition using the same argument we deduce that vn 6= 0. Thus vi 6= 0 for all
i ∈ {1, . . . , n}. According to Theorem 2.12, together with the fact that {v1 , . . . , vn } are
orthogonal, we conclude that {v1 , . . . , vn } are linearly independent forming an orthogonal
basis of U .
Step 1. We start with any basis {u1 , . . . , un } of U and apply the Gram-Schmidt
process to this basis. If (vk , vk ) = 0 for some 1 ≤ k ≤ n then (·, ·) is not an inner product
since according to Theorem 2.29 if it is an inner product vk 6= 0 for all k = 1, . . . , n.
Step 2. Suppose that (vk , vk ) 6= 0 for all k = 1, . . . , n, so the Gram-Schmidt does get
to the end. According to Theorem 2.28 we get an orthogonal basis {v1 , . . . , vn } of U .
• If (vk , vk ) < 0 for some k ∈ {1, . . . , n} then (·, ·) is not positive definite and obviously
is not an inner product.
• If (vk , vk ) > 0 for all k ∈ {1, . . . , n} then according to Theorem 2.18, (·, ·) is an
inner product.
Example 2.30. Let U = R2 and consider the symmetric bilinear form on U associated
with the matrix
1 2
G= ,
2 1
i.e., for u = (x1 , x2 )T ∈ U and v = (y1 , y2 )T ∈ U , we have
We want to check whether (·, ·)G is an inner product on U . We apply the Gram-Schmidt
to the standard basis {e1 , e2 } of R2 .
1 (e2 , v1 )G
v1 = e 1 = , v2 = e2 − v1 .
0 (v1 , v1 )G
61
2.6. GRAM-SCHMIDT ORTHOGONALISATION
Example 2.31. Consider U = R2 equipped with the standard inner product. Let
2 1
u1 = , u 2 = .
1 1
Then (u1 , u2 ) is a non-orthogonal basis of U . We apply the Gram-Schmidt process to these
vectors
2 (u2 , v1 ) 1 3 2 −1/5
v1 = u1 = , v2 = u2 − v1 = − = .
1 (v1 , v1 ) 1 5 1 2/5
Then {v1 , v2 } is an orthogonal basis of U which can be directly verified.
Example 2.32. As in Examples 2.10 and 2.16, we consider the inner product on U =
P2 (R) given by Z 1
(u, v) = u(x)v(x) dx.
−1
In Example 2.16 we already proved that (·, ·) is an inner product. Using the computations
in Example 2.10, we now apply the Gram-Schmidt to the standard basis {1, x, x2 } of
U:
v1 = 1,
(x, 1) 0
v2 = x − 1 = x − 1 = x,
(1, 1) 2
2 2
(x , x) (x , 1) 0 2/3 1
v3 = x2 − x− 1 = x2 − x− 1 = x2 − .
(x, x) (1, 1) 2/3 2 3
Then {v1 , v2 , v3 } form an orthogonal basis of U = P2 (R).
62
2.6. GRAM-SCHMIDT ORTHOGONALISATION
v1 = 1, v2 = x, v3 = x2 − 1/3,
Hence we obtain an orthonormal basis for the above inner product space, which is given
by
n v
1 v2 v3 o n 1 x x2 − 1/3 o
, , = √ ,p , p .
kv1 k kv2 k kv3 k 2 2/3 8/45
63
2.6. GRAM-SCHMIDT ORTHOGONALISATION
Moreover,
n 2
2
X (u, ui )
kuk = .
i=1
kui k
n n n X
n n n 2
2
X X X X X (u, ui )
kuk = (u, u) = ai u i , aj u j = ai aj (ui , uj ) = a2i kui k2 = .
i=1 j=1 i=1 j=1 i=1 i=1
kui k
Moreover,
n
X
2
kuk = (u, ui )2 .
i=1
64
2.6. GRAM-SCHMIDT ORTHOGONALISATION
Example 2.37. Let U = Rn be the Euclidean vector space with the standard inner product
and let B = {e1 , . . . , en } be the standard basis. Since
1, if i = j
(ei , ej ) =
0, if i 6= j
Theorem 2.36 simply generalises this example to an orthonormal basis for a general inner
product space.
1 x x2 − 1/3
u1 = √ , u2 = p , u3 = p ,
2 2/3 8/45
u = a1 u1 + a2 u2 + a3 u3 ,
where
Z 1
1 28
a1 = (u1 , u) = √ (2x2 − 3x + 4) dx = √ ,
−1 2 3 2
Z 1
x 2 √
a2 = (u2 , u) = p (2x2 − 3x + 4) dx = − p = − 6,
−1 2/3 2/3
1 2
x − 1/3
Z
16 8
a3 = (u3 , u) = p (2x2 − 3x + 4) = p =√ .
−1 8/45 45 8/45 90
65
Chapter 3
This chapter is the beginning of the second part of this lecture notes that is about
Linear Programming (LP). The chapter provides key concepts of LP that include its
definition and terminologies, standard and cannonical forms. Some typically important
realistic problems that can be modelled and solved using LP are also discussed. The
following topics will be covered:
• definition of LP,
• terminologies of LP,
• LP modelling.
• recognise and formulate some typical problems in real applications into LPs.
[BT97, Chapter 1] and [DT97, Chapter 1] discuss similar topics as in this chapter that
you may find useful.
66
3.1. WHAT IS LINEAR PROGRAMMING?
A good understanding of the theory and algorithms for linear programming is essential
for understanding
• game theory,
So, the theory of LP serves as a guide and motivating force for developing results for other
mathematical optimization problems with either continuous or discrete variables.
Example 3.1. A small machine shop manufactures two models: standard and deluxe.
• The manufacturer has three grinders and two polishers. Therefore in 40 hours week
there are 120 hours of grinding capacity and 80 hours of polishing capacity.
• There is a net profit of £3 on each standard model and £4 on each deluxe model.
67
3.2. DEFINITION OF LP:
To maximize the profit, the manager must decide on the allocation of the available pro-
duction capacity to standard and deluxe models.
Let x1 and x2 be the numbers of standard and deluxe models manufactured, respectively.
For this small machine shop, there are two constraints (restrictions):
• Grinding restriction:
2x1 + 5x2 ≤ 120.
• Polishing restriction:
4x1 + 2x2 ≤ 80.
x1 ≥ 0, x2 ≥ 0. (W hy?)
68
3.3. TERMINOLOGY
xj ≥ 0, j = 1, ..., n.
Minimize (Maximize) cT x
subject to Ax (≥, =, ≤) b,
x ≥ 0,
where
a a12 · · · a1n b1 c1 x1
11
a21 a22 · · ·
a2n b2 c2 x2
..
A= .. .. .. .. ..
, b = , c = ,x = .
. . . . . .
am1 am2 · · · amn bm cn xn
All vectors are understood as column vectors, unless otherwise stated.
A = [a1 , a2 , ..., an ]
3.3 Terminology
• The function
cT x = c1 x1 + ... + cn xn
69
3.3. TERMINOLOGY
• The inequality ‘Ax ≤ b’ is called the constraint, and aij xj ≤ bi denotes the
Pn
j=1
ith constraint.
• The set of all such points is called the feasible region (or feasible set).
Example 3.3.
Minimize 2x1 + x2
subject to x1 + x2 ≤ 8,
x1 ≥ 0, x2 ≥ 0.
For this example, the problem has two decision variables x1 and x2 . The objective function
to be minimized is 2x1 + x2 . The LP problem is to find a point in the feasible region with
the smallest objective value.
70
3.4. STANDARD FORMS OF LPS:
Minimize cT x
subject to Ax = b,
x ≥ 0.
Throughout this course, we call the above minimization problem as a standard form. In
fact, any LP problem can be converted to this form.
dT x ≥ α ⇐⇒ dT x − t = α, t ≥ 0.
In matrix form:
Ax ≤ b ⇐⇒ Ax + s = b, s ≥ 0.
So, by multiplying the coefficients of the objective function by −1, the maximization
problem can be converted into a minimization problem.
Lj ≤ xj ≤ Uj ⇐⇒ xj ≤ Uj and xj ≥ Lj
71
3.5. CANONICAL FORM OF LPS
Example 3.4. Write down the standard form of the following linear programming:
max 4x1 + x2 − x3
s.t. x1 + 3x3 ≤ 6,
3x1 + x2 + 3x3 ≥ 9,
x1 , x2 ≥ 0, x3 is unrestricted in sign.
Example 3.5. Write down the standard form of the following linear programming:
Ťhe LP problem
min cT x
s.t. Ax ≥ b,
x≥0
is called the canonical form of the LP.
Ľater, we will see that the canonical form is very useful in the duality representation
of LPs. Introducing a slack vector, the canonical form can be easily converted to the
standard form
min cT x
s.t. Ax − s = b
x ≥ 0, s ≥ 0.
Čonversely, a standard from can be easily transformed to a canonical form. (Do this
exercise)
3.6 LP Modeling
72
3.6. LP MODELING
• A manufacturing firm has to decide on the mix of Ipads and IPhones to be produced.
• A market research indicated that at most 1000 units of Ipads and 4000 units of
IPhones can be sold per month.
• The unit profits of the Ipads and IPhones are £300 and £400 respectively.
It is desired to find the number of unites of Ipads and IPhones that the firm must produce
in order to maximize its profit.
Determine the decision variables. Suppose that the number of the units for Ipads is x1
and the number of the units for IPhones is x2 .
Determine the explicit constraints. There are two kinds of constraints: The market
constraints and man-hours restriction. For the former, we have
x1 ≤ 1000,
x2 ≤ 4000.
73
3.6. LP MODELING
300x1 + 400x2 .
Determine the implicit constraints. The number of Ipads and IPhones cannot be neg-
ative, so x1 , x2 ≥ 0.
x1 ≤ 1000,
x2 ≤ 4000,
x1 , x2 ≥ 0.
Example 3.7 (The Workforce Planning Problem). Čonsider a restaurant that is open
seven days a week. Based on past experience, the number of workers needed on a particular
day is given as follows:
Ěvery worker works five consecutive days, and then takes two days off, repeating this
pattern indefinitely. How can we minimize the number of workers that staff the restaurant?
Define the variables: Let the days be numbers 1 through 7 and let xi be the number
of workers who begin their five-day shift on day i.
Ľet us consider Monday as an example. People work on Monday must start to work
on Thur, Fri, Sat, Sun and Monday. Thus the total number of workers on Monday is
x1 + x4 + x5 + x6 + x7
74
3.6. LP MODELING
x1 + x4 + x5 + x6 + x7 ≥ 14.
Similarly, we may consider other days. Thus, the linear programming model for this
problem is given as follow.
7
min
X
xi
i=1
s.t x1 + x4 + x5 + x6 + x7 ≥ 14 (M on),
x1 + x2 + x5 + x6 + x7 ≥ 13 (T ue),
x1 + x2 + x3 + x6 + x7 ≥ 15 (W ed),
x1 + x2 + x3 + x4 + x7 ≥ 16 (T hur),
x1 + x2 + x3 + x4 + x5 ≥ 19 (F ri),
x2 + x3 + x4 + x5 + x6 ≥ 18 (Sat),
x3 + x4 + x5 + x6 + x7 ≥ 11 (Sun),
xi ≥ 0, i = 1, ..., 7.
• There are m different types of food, F1 , . . . , Fm , that supply varying quantities of the
n nutrients, N1 , . . . , Nn , that are essential to good health.
b1 y1 + b2 y2 + · · · + bm ym . (3.1)
75
3.6. LP MODELING
for j = 1, . . . , n.
W̌e need to ensure that all the minimum daily requirements are met, that is,
y1 ≥ 0, y2 ≥ 0, . . . , ym ≥ 0. (3.3)
Ťhe problem is to minimize (3.1) subject to (3.2) and (3.3). This is exactly the fol-
lowing LP problem:
Minimize b1 y1 + b2 y2 + · · · + bm ym
s.t a1j y1 + a2j y2 + · · · + amj ym ≥ cj , j = 1, . . . , n,
y1 ≥ 0, y2 ≥ 0, . . . , ym ≥ 0.
• There are I production plants, P1 , ..., PI , that supply a certain commodity, and there
are J markets, M1 , . . . , MJ , to which this commodity must be shipped.
• Let cij be the cost of transporting one unit of the commodity from plant Pi to market
Mj .
1) Let xij be the quantity of the commodity shipped from plant Pi to market Mj .
76
3.6. LP MODELING
j=1
Ťhe problem is to minimize the objective (3.4) subject to the constraints (3.5), (3.6)
and (3.7), i.e.,
I X
J
Minimize
X
xij cij
i=1 j=1
J
X
s.t xij ≤ si , i = 1, . . . , I,
j=1
I
X
xij ≥ rj , j = 1, . . . , J,
i=1
xij ≥ 0, i = 1, . . . , I, j = 1, . . . , J.
77
3.6. LP MODELING
• Let us suppose that only one people is allowed on a job at a time (two people cannot
work on the same job at the same time).
The problem is to choose an assignment of persons to jobs to maximize the total value.
100
(3.9)
X
xij ≤ 1, j = 1, . . . , 10,
i=1
and
xij ≥ 0, i = 1, . . . , 100, j = 1, . . . , 10. (3.10)
• Inequality (3.8) reflects the fact that a person cannot spend more than 100 percentage
of his time working.
• Inequality (3.9) means that the total time spent on a job is not allowed to exceed a
day.
• (3.10) says that no one can work a negative amount of time on any job.
Subject to constraints (3.8), (3.9) and (3.10), we wish to maximize the total value,
100 X
10
(3.11)
X
aij xij .
i=1 j=1
78
3.7. A BRIEF HISTORY OF LPS
Ľinear programming was developed during the second world war to plan expenditures
and returns in order to reduce costs to the army and increase losses to the enemy.
Ǐt was kept secret until 1947. Postwar, many industries found its use in their daily
planning.
• John von Neumann developed the theory of the duality in the same year.
• The linear programming problem was first shown to be solvable in polynomial time
by Leonid Khachiyan in 1979.
• A larger theoretical and practical breakthrough in the field came in 1984 when
Narendra Karmarkar introduced a new interior point method for solving linear
programming problems.
79
Chapter 4
In this chapter, we will study LPs with 2 variables and how to solve them using the
geometric method. Topics that will be covered:
At the end of this chapter students should be able to use geometric method to solve a
two variable LP problem determining whether it is infeasible, unbounded or has a finite
solution. [DT97, Chapter 2] presents similar topics to this chapter that you may find
helpful.
80
The following result contains the underlying idea of the geometric method for solving
LPs with two variables that we will study in this chapter.
Geometric solution:
• When the feasible region is unbounded, we might move the plane indefinitely while
always intersecting the feasible region, and hence the problem is unbounded.
• For a bounded and some unbounded feasible regions, the objective hyperplane moves
and must stop at a certain point, otherwise it goes beyond the feasible region. In
such cases, the problem has a finite optimal solution.
• The feasible set might be empty, and hence the problem is infeasible.
Example 4.3 (Finite solution case). Solve the LP geometrically (or graphically)
Minimize x1 + x2
subject to 2x1 + x2 ≤ 6,
−x1 + x2 ≤ 4,
x1 , x2 ≥ 0.
81
1 . Sketch the feasible set
2 . Pass one point in feasible set and draw the objective hyperplane
x2
-x1+x2=4
Objective
hyper plane
(0,0) x1 (0,0)
2x1+x2=6
-c
Figure 4.1: Figure for Example 4.3: In the left picture, the feasible domain is the shaded
region formed by the two lines x1 + x2 = 6 and −x1 + x2 = 4 as well as the two axes.
In the right picture, we move the objective hyperplane x1 + x2 along the direction of the
negative of the cost coefficient vector until reaching the optimal point (0, 0).
Maximize x1 + x2
subject to −x1 + x2 ≤ 4,
x1 , x2 ≥ 0.
82
Štill, we follow the above three steps and it is easy to see from Figure 4.2 that that this
problem has an unbounded solution (no finite optimal solution).
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-x1+x2=4
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x2 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Unbounded
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
feasible region
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx c
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
x1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx c
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Objective function
Figure 4.2: Example 4.4: The feasible region, which is formed by the line −x1 + x2 = 4
and the two axes, is unbounded. Moving the objective hyperplane along the direction of
the cost coefficient vector will increase the objective function infinitely. Thus the problem
has an unbounded solution.
Minimize x1 + x2
subject to −x1 + x2 ≥ 4,
x1 − 2x2 ≥ 6,
x1 , x2 ≥ 0.
1 . The problem has a finite optimal solution (unique optimal solution or infinitely
many optimal solutions).
83
3 . The problem is infeasible, i.e., the feasible region is empty.
Proposition 4.6. If a linear programming problem has a finite optimal solution, then it
has either a unique solution or infinitely many solutions.
min cT x
s.t Ax = b,
x ≥ 0.
Suppose that the optimal solution of the LP is not unique. Then it has at least two
different optimal solutions, denoted by x
b and x̄.
Since x
b and x̄ are optimal solutions, they must be feasible to the problem, and the
objective values at these two solutions are equal, i.e.
Ax̂ = b, x̂ ≥ 0, Ax̄ = b, x̄ ≥ 0
and
cT x̂ = cT x̄ = z ∗ ,
where z ∗ denotes the optimal objective value. Now, let’s consider the point of the form
= αAx̂ + (1 − α)Ax̄
= αb + (1 − α)b
= b.
Therefore, x(α) is a feasible solution to the linear programming problem. Finally, checking
84
the function value at x(α), we have
= αcT x̂ + (1 − α)cT x̄
= αz ∗ + (1 − α)z ∗
= z∗,
Therefore, x(α) is an optimal solution of the LP. Since α can be any number in (0,1),
all x(α) given by (4.1) are optimal to the LP, and hence the LP problem has infinitely
many optimal solutions.
85
Chapter 5
In this chapter, we study basic knowledge in convex analysis that will be used in
subsequent chapters. The following topics will be covered
• representation of polyhedra.
[BT97, Chapter 2] contains similar material to this chapter that you may find useful.
{x : pT x = α}
where p is a nonzero vector called the normal or the gradient to the hyperplane.
86
5.1. BASIC CONCEPTS IN CONVEX ANALYSIS
{x : pT x ≥ α} or {x : pT x ≤ α}.
H = {x : pT x = α},
H + = {x : pT x ≤ α},
H − = {x : pT x ≥ α}.
Convex combination
α + β + γ = 1,
α, β, γ ≥ 0.
87
5.1. BASIC CONCEPTS IN CONVEX ANALYSIS
1 1 1
b = u1 + u2 + u3 .
2 3 6
P = {x : Ax ≤ b},
Convex set
(ii) Since each half-space is convex, and a polyhedron is the intersection of a finite
number of half-spaces, so a polyhedron is convex.
be the intersection of Si ’s. We now prove that S is a convex set. Indeed, let x, y ∈ S.
We now prove that for any α ∈ (0, 1), we have x(α) = αx + (1 − α)y ∈ S. In fact, since
88
5.1. BASIC CONCEPTS IN CONVEX ANALYSIS
x, y ∈ S which is the intersection of Si ’s, the vectors x and y are in Si for all i = 1, ..., n.
By the convexity of Si , we must have
E = {x : (x − y)T Q(x − y) ≤ 1}
P = {x : Ax ≤ 0}
Proof. If the set can be represented as P = {x : Ax ≤ 0} for some matrix A, then for any
x ∈ P we have αx ∈ P for any α ≥ 0. Thus, P is a cone.
We now further assume that P is a cone. By the definition of cone, for any x ∈ P , it
must hold that αx ∈ P for any α ≥ 0. In particular, by taking α = 0, we see that 0 ∈ P ,
89
5.1. BASIC CONCEPTS IN CONVEX ANALYSIS
thus b ≥ 0. Let α > 0 be any sufficiently large number, it follows from A(αx) ≤ b that
Ax ≤ 0. Thus P ⊆ {x : Ax ≤ 0}. Since b ≥ 0, it further implies that
P ⊆ {x : Ax ≤ 0} ⊆ P,
so P = {x : Ax ≤ 0}.
Convex function
Proof. This can be verified directly. Let x, y ∈ Rd and λ ∈ [0, 1], then
By subtracting the left-hand side from the right-hand side, the above equality is equivalent
to
λ(1 − λ)(x − y)2 ≥ 0,
90
5.2. EXTREME POINTS, DIRECTIONS, REPRESENTATION OF POLYHEDRA
In summary,
Definition 5.10 (Extreme point). A point x in a convex set X is called an extreme point
of X, if x cannot be represented as a strict convex combination of two distinct points in
X. In other words, if x = λx1 + (1 − λ)x2 with 0 < λ < 1 and x1 , x2 ∈ X then it must
have x = x1 = x2 .
Definition 5.11 (Rays). Let d 6= 0 be a vector and x0 be a point. The collection of points
of the form {x0 + λd : λ ≥ 0} is called a ray.
x0 is called the vertex of the ray, and d is the direction of the ray.
Therefore, starting at any point x0 in the set, one can recede along d for any step
length λ ≥ 0 and remain within the set.
Example 5.13.
91
5.2. EXTREME POINTS, DIRECTIONS, REPRESENTATION OF POLYHEDRA
What is the direction of X? Ľet x0 = (x1 , x2 )T be an arbitrary fixed feasible point. Then
d = (d1 , d2 )T is a direction of X if and only if
(d1 , d2 )T 6= 0
and
x0 + λd = (x1 + λd1 , x2 + λd2 )T ∈ X, ∀λ ≥ 0.
Therefore,
x1 − x2 + λ(d1 − d2 ) ≥ −2,
x1 + λd1 ≥ 0,
x2 + λd2 ≥ 1,
for all λ ≥ 0. Since the last two inequalities must hold for fixed x1 and x2 and for all
λ ≥ 0, we conclude that d1 , d2 ≥ 0. Similarly, from the first two inequalities above, we
conclude that
d1 − 2d2 ≥ 0, d1 − d2 ≥ 0.
(d1 , d2 )T 6= 0,
d1 ≥ 0, d2 ≥ 0, d1 ≥ 2d2 .
Example 5.16. 1. What is the extreme direction of the set X in Example 5.14?
92
5.2. EXTREME POINTS, DIRECTIONS, REPRESENTATION OF POLYHEDRA
P = {x ∈ Rn : Ax ≤ b} 6= ∅
Proof. This theorem can be proved directly using convex analysis. However, that ap-
proach is beyond the scope of this course. Instead, as will be seen in Theorem 7.5, the
set of extreme points of a polyhedron is the same as the set of basic feasible solution of
a linear programming problem defined on the polyhedron. Thus the number of extreme
points is the same as the number of basic feasible solution, which is finite (cf. Section
7.1.4).
min{cT x : x ∈ P }
has a finite optimal solution, then there is an optimal solution that is an extreme point.
Proof. Similarly as Theorem 5.17, this theorem can also be proved directly using convex
analysis. However, it can be seen as a direct consequences of Theorem 7.5 and Theorem
7.6 in Chapter 7.
d ≥ 0, d 6= 0, and Ad ≤ 0.
D = {d : Ad ≤ 0, eT d = 1, d ≥ 0}.
93
5.2. EXTREME POINTS, DIRECTIONS, REPRESENTATION OF POLYHEDRA
2). Similarly, we may prove that the extreme directions of the nonempty polyhedral
set
X = {x : Ax = b, x ≥ 0}
D = {d : Ad = 0, eT d = 1, d ≥ 0}.
n k
X l
X k
X
j i
P = x: x= λj x + µi d , λj = 1,
j=1 i=1 j=1
o
λj ≥ 0, j = 1, ..., k, µi ≥ 0, i = 1, ..., l ,
where {x1 , x2 , ..., xk } is the set of extreme points of P and {d1 , d2 , ..., dl } is the set of
extreme rays of P .
94
Chapter 6
Duality theory
In this chapter we will study an important topic of LP, namely duality theory. Duality
in linear programming is essentially a unifying theory that develops the relationships
between a given LP problem (the primal problem) and another related LP problem (the
dual problem). We also study the optimality condition, i.e., the so-called Karush-Kuhn-
Tucker (KKT) optimality condition, for LP problems. The duality theory and optimality
conditions form the backbone of linear and nonlinear optimization problems.
• optimality conditions,
95
6.1. MOTIVATION AND DERIVATION OF DUAL LP PROBLEMS
[BT97, Chapter 5] and [DT97, Chapter 5] contain relevant topics to this chapter that you
may find helpful.
Example 6.1 (minimising cost vs. maximising profit). There are two products, A and
B, that are made up from two elements, e and f . Product A contains 5 units of element
e and 2 units of element f and can be sold with price 16 Yuan, while product B contains
3 units of element e and 4 units of element f and can be sold with price 10 Yuan.
Let x and y be the units of product A and B that the customer buys. The above
problem can be formulated into a minimising LP problem as follows
subject to
Minimise cT x̄,
subject to
Ax̄ ≥ b,
x̄ ≥ 0,
96
6.1. MOTIVATION AND DERIVATION OF DUAL LP PROBLEMS
where
x 5 3 100 16
x̄ = , A= , b = , c = .
y 2 4 60 10
Production’s perspective. Now the supplier wants to assign price for each unit of
element e and f so that the customer’s need can be met and the profit is maximal. Let u
and v be the price that the supplier should assign to each unit of element e and f . His/her
task can be formulated into the following maximising LP problem
subject to
We also can write this LP in a matrix form using the notations above as
Maximise bT ȳ,
subject to
AT ȳ ≤ c,
ȳ ≥ 0,
Suppose now in Example above we only know the minimization problem. How do we
attempt to solve this? Since we are looking for a minimum value of the objective function,
we first need to find a lower bound for it. To do so, we combine the constraints to obtain
a lower bound of the type
u(5x + 3y) + v(2x + 4y) = (5u + 2v)x + (3u + 4v)y ≥ 100u + 60v, (6.1)
5u + 2v ≤ 16 and 3u + 4v ≤ 10.
97
6.2. PRIMAL AND DUAL PROBLEMS OF LPS
Finally to achieve the best optimal lower bound for the objective function, we should
maximise the lower bound in (6.1)
subject to
5u + 2v ≤ 16,
3u + 4v ≤ 10,
u, v ≥ 0
This is exactly the problem from the production’s perspective in Example 6.1.
Duality deals with pairs of LPs and the relationship between their solutions. One
problem is called primal and the other the dual.
It does not matter which problem is called the primal since the dual of the dual is the
primal (see Proposition 6.5 below).
Primal Problem:
(LP ) Minimize cT x
s.t. Ax = b, x ≥ 0.
Dual problem:
(DP ) Maximize bT y
s.t. AT y ≤ c,
where y ∈ Rm .
98
6.2. PRIMAL AND DUAL PROBLEMS OF LPS
(DP ) Maximize bT y
s.t. AT y + s = c, s ≥ 0
s.t. 3x1 + x2 − x3 = 4,
5x1 + 2x2 − x4 = 7,
x1 , x2 , x3 , x4 ≥ 0.
w1 + 2w2 ≤ 8
−w1 ≤ 0
−w2 ≤ 0.
When an LP is not given in standard form, we may first convert them to the standard
form, and then write down their dual problems.
s.t. 3x1 + x2 − x3 ≥ 4,
5x1 + 2x2 − x4 ≤ 7,
x1 , x2 , x3 ≥ 0, x4 is free.
(LP ) Minimize cT x
s.t. Ax ≥ b, x ≥ 0,
99
6.2. PRIMAL AND DUAL PROBLEMS OF LPS
(DP ) Maximize bT y
s.t. AT y ≤ c, y ≥ 0.
s.t. 3x1 + x2 − x3 ≥ 4,
5x1 + 2x2 − x4 ≤ 7,
x1 , x2 , , x3 , x4 ≥ 0.
(LP ) Minimize cT x
s.t. Ax = b, x ≥ 0.
(DP ) Maximize bT y
s.t. AT y + s = c, s ≥ 0.
We now consider the dual of (DP). First we transform this problem as the standard form.
Thus we have
s.t. AT y 0 − AT y 00 + s = c, s ≥ 0, y 0 ≥ 0, y 00 ≥ 0.
100
6.3. DUALITY THEOREM
(DDP ) Maximize cT z
−b
s.t. T T T
(A , −A , I) z ≤ b ,
0
(DDP ) Maximize cT z
It is equivalent to
s.t. −Az = b, z ≤ 0.
(DDP ) Minimize cT w
s.t. Aw = b, w ≥ 0.
Denote by Fp and Fd the feasible regions of primal and dual problems, respectively,
i.e.
Fp = {x : Ax = b, x ≥ 0},
Fd = {(y, s) : AT y + s = c, s ≥ 0}.
The following result shows that the objective value at any primal feasible solution is at
least as large as the objective value at any feasible dual solution.
Theorem 6.7 (Weak duality Theorem). Let x be any feasible point of the primal problem,
i.e. x ∈ Fp and y be any feasible point of the dual problem, i.e., (y, s) ∈ Fd . Then
cT x ≥ bT y.
101
6.3. DUALITY THEOREM
Proof. For any feasible point x ∈ Fd and (y, s) ∈ Fd , we have Ax = b and x ≥ 0, and
s = c − AT y ≥ 0, and thus
cT x − bT y = cT x − (Ax)T y = xT (c − AT y) = xT s ≥ 0.
The last inequality above follows from the fact that (x, s) ≥ 0.
The quantity cT x − bT y is often called the duality gap. From the weak duality
theorem we have the following immediate consequences:
Corollary 6.8. If the primal LP is unbounded (i.e., cT x → −∞) , then the dual LP must
be infeasible.
Corollary 6.9. If the dual LP is unbounded, then the primal LP must be infeasible.
Thus, if either problem has an unbounded objective value, then the other problem
possesses no feasible solution.
Corollary 6.10. If x is feasible to the primal LP, y is feasible to the dual LP, and
cT x = bT y, then x must be optimal to the primal LP and y must be optimal to the dual
LP.
Minimize −x1 − x2
s.t. x1 − x2 ≥ 1,
−x1 + x2 ≥ 1,
x1 , x2 ≥ 0.
102
6.3. DUALITY THEOREM
Maximize w1 + w2
s.t. w1 − w2 ≤ −1,
−w1 + w2 ≤ −1,
w1 , w2 ≥ 0.
The last corollary above identifies a sufficient condition for the optimality of a primal-
dual pair of feasible solutions, namely that their objective values coincide. One natural
question to ask is whether this is a necessary condition. The answer is yes, as shown by
the next result.
Theorem 6.12 (Strong Duality Theorem). If both the primal LP and the dual LP have
feasible solutions, then they both have optimal solutions, and for any primal optimal
solution x and dual optimal solution y we have that cT x = bT y.
Proof. As will be seen in Chapter 11 one can solve both the primal and the dual problem
simultaneously using the simplex method.
• One problem has unbounded objective value, in which case the other problem must
be infeasible.
In Summary,
P optimal ⇔ D optimal
P unbounded ⇒ D infeasible
103
6.4. NETWORK FLOW PROBLEMS
D unbounded ⇒ P infeasible
Definition 6.13. A network N = (V, E) is a directed graph where V and E are respectively
the set of vertices (nodes) and edges (arcs). Given two vertices i, j ∈ V we denote the
edge between them by (i, j).
Definition 6.14. The capacity of an edge is the maximum amount of flow that can pass
through an edge in both directions. Formally it is a map c : E → R+ , called the capacity
map.
In most networks there are one or more nodes which are distinguished. A source is a
node such that all edges through it are oriented away from it. Similarly a sink is a node
such that all edges through it are oriented towards it.
• capacity constraint: The flow of an edge cannot exceed its capacity, in other words:
fij ≤ cij , for each (i, j) ∈ E;
104
6.4. NETWORK FLOW PROBLEMS
• conservation of flows: The sum of the flows entering a node must equal the sum
of the flows exiting that node, except for the sources and the sinks:
P
i:(i,j)∈E fij =
Example 6.16 (The max-flow problem). Let N = (V, E) be a directed network with only
one source S ∈ V , only one sink T ∈ V and a set of intermediate nodes. Assume the flow
is never negative, namely if along an edge (i, j) the flow fij is strictly positive, then the
flow fji in the opposite direction is 0. The maximal flow problem is to maximise the flow
from source to sink subject to the network edge capacities and conservation of flow:
Maximise
X
fSj
i:(S,j)∈E
In the context of flow analysis, there is only an interest in considering how units are
transferred between source and sink. Namely, we can re-write the above linear problem by
considering the whole paths from S to T . Let P be the set of all possible paths from S to
T . We associate with each path p ∈ P a quantity xp specifying how much of the flow from
S to T is being transferred along the path. With this view, the conservation constraint
above is unnecessary and we obtain the following simpler linear programming problem
Maximise
X
xp
p∈P
p∈P :(i,j)∈p
xp ≥ 0, for all p ∈ P.
Example 6.17 (The min-cut problem). Let N = (V, E) be a directed network with only
one source S ∈ V , only one sink T ∈ V and a set of intermediate nodes. Let yij denote
the weight to assign to an edge (i, j). The problem is to separate S and T with minimum
total weighted capacity. The condition that source and sink are separated can be stated as
follows: for every path p ∈ P from the source S to the sink T , the weight of the path is at
105
6.4. NETWORK FLOW PROBLEMS
least 1.
Minimise
X
cij yij
(i,j)∈E
subject to
X
yij ≥ 1 ∀p ∈ P
(i,j)∈p
yij ≥ 0 (i, j) ∈ E.
The max-flow min-cut theorem is a special case of the duality theorem. It states that
in a flow network, the maximum amount of flow passing from the source to the sink is
equal to the total weight of the edges in the minimum cut. The max-flow min-cut theorem
has many practical applications in networks, transportation and scheduling.
Example 6.18 (Example of the max-flow min-cut theorem). We consider the following
network
Figure 6.1: Example of the max-flow min-cut theorem. The network consists of 5 nodes
{S, A, B, C, T } including the source S and the sink T . The arrow indicates the direction
of flow in each edge, and the number beside each edge is the capacity of that edge.
In this example
• E = {edges} = {(S, A), (S, B), (A, C), (A, T ), (B, T ), (C, T )}.
106
6.4. NETWORK FLOW PROBLEMS
Maximise
X
xp
p∈P
p∈P :(i,j)∈p
xp ≥ 0, for all p ∈ P.
In the above problem, P denotes the set of all paths from the source S to the sink T . In
this example, there are three paths from S to T
Let x1 , x2 , x3 be the flow transferred along the corresponding path. The max-flow problem
for this example is
Maximise x1 + x2 + x3
x1 ≤ 3 (for (A,C)),
x2 ≤ 4 (for (A,T)),
x3 ≤ 7 (for (S,B)),
x3 ≤ 6 (for (B,T)),
x1 ≤ 5 (for (C,T)),
x1 , x2 , x3 ≥ 0.
x1 + x2 + x3 = (x1 + x2 ) + x3 ≤ 2 + 6 = 8
Minimise
X
cij yij
(i,j)∈E
subject to
X
yij ≥ 1 ∀p ∈ P
(i,j)∈p
yij ≥ 0 (i, j) ∈ E.
107
6.5. OPTIMALITY CONDITIONS & COMPLEMENTARY SLACKNESS
subject to y1 + y3 + y6 ≥ 1 (for p1 ),
y1 + y4 ≥ 1 (for p2 ),
y2 + y5 ≥ 1 (for p3 ),
y1 , y2 , y3 , y4 , y5 , y6 ≥ 0.
2y1 +7y2 +3y3 +4y4 +6y5 +5y6 = (y1 +y3 +y6 )+(y1 +y4 )+6(y2 +y5 )+y2 +2y3 +3y4 +4y6 ≥ 8,
We now interpret the result from the min-cut problem. Recalling that a cut is simply a
partition of the nodes V into two disjoint parts V = V1 ∪ V2 such that S ∈ V1 and T ∈ V2 .
The capacity of a cut is the sum of the capacity of edges going out the cut. Examples of
cuts with their capacities are
The cut with minimum capacity is {S, B} ∪ {A, C, T } whose capacity is 8 = 2 + 6 which
is the same as the optimal max flow.
The optimality condition can be obtained by the strong duality theorem from the last
lecture.
The strong duality theorem provides the condition to identify optimal solutions:
108
6.5. OPTIMALITY CONDITIONS & COMPLEMENTARY SLACKNESS
Minimize cT x
s.t. Ax = b, x ≥ 0,
AT ȳ ≤ c, (6.2)
Ax̄ = b, x̄ ≥ 0, (6.3)
bT ȳ = cT x̄. (6.4)
Remark. Similarly, the optimality condition for the duality problem is the same as
above.
AT ȳ ≤ c
Ax̄ = b, x̄ ≥ 0,
bT ȳ = cT x̄
AT ȳ + s̄ = c, s̄ ≥ 0
Ax̄ = b, x̄ ≥ 0,
x̄T s̄ = 0.
109
6.5. OPTIMALITY CONDITIONS & COMPLEMENTARY SLACKNESS
Proof. First, adding the slack variable vector s̄ to inequality (6.2), we have
AT ȳ + s̄ = c, s̄ ≥ 0
x̄T s̄ = 0.
Minimize cT x
s.t. Ax = b, x ≥ 0
AT ȳ + s̄ = c, s̄ ≥ 0,
Ax̄ = b, x̄ ≥ 0,
x̄T s̄ = 0.
Minimize −2x1 − x2
−x1 − x2 ≥ −20,
x1 , x2 ≥ 0.
Verify that whether (15,0) and (20, 0) are optimal solutions to the problem.
110
6.5. OPTIMALITY CONDITIONS & COMPLEMENTARY SLACKNESS
• Introduce slack variables x3 and x4 and transform the problem to the standard form
Minimize −2x1 − x2
−x1 − x2 − x4 = −20,
x1 , x2 , x3 , x4 ≥ 0.
x1 + 2x2 − x3 = 15,
−x1 − x2 − x4 = −20,
x1 , x2 , x3 , x4 ≥ 0,
y1 − y2 ≤ −2,
2y1 − y2 ≤ −1,
−y1 ≤ 0, −y2 ≤ 0,
• Consider the point (x1 , x2 ) = (15, 0). From the first two equalities above, we see that
(x3 , x4 ) = (0, 5) satisfies the first three conditions. Thus at x = (15, 0, 0, 5)T the
above optimality conditions are reduced to
y1 − y2 ≤ −2
2y1 − y2 ≤ −1
−y1 ≤ 0, −y2 ≤ 0
It is easy to very that this system is inconsistent (there exists no y satisfies these
conditions). Thus, (x1 , x2 ) = (15, 0) is not an optimal solution to the LP.
• Consider the point (x1 , x2 ) = (20, 0). Clearly, this point, together with (x3 , x4 ) =
(5, 0), satisfies the first three conditions of the optimality. We now check if there
111
6.6. COMPLEMENTARY SLACKNESS CONDITION
y1 − y2 ≤ −2
2y1 − y2 ≤ −1
−y1 ≤ 0, −y2 ≤ 0
Clearly, (y1 , y2 ) = (0, 2) satisfies these conditions, thus the point (x1 , x2 ) = (20, 0)
is an optimal solution to the LP problem.
From the optimality condition (see Theorem 6.21), we note that at the optimal so-
lution x∗ and (y ∗ , s∗ ), we have (x∗ )T s∗ = 0, which is equivalent to x∗j s∗j = 0 for every
j = 1, ..., n since the nonnegativity of x∗ and s∗ . Thus, at the optimal solution, one of the
terms x∗j and s∗j must be zero. This is called complementary slackness condition.
Consider the LP
Minimize cT x
s.t. Ax = b, x ≥ 0,
AT ȳ + s̄ = c, s̄ ≥ 0. (Dual feasibility)
Theorem 6.23. x and (y, s) are optimal solutions to primal and dual problems, respec-
tively, if and only if they satisfy the primal feasibility, dual feasibility and the complemen-
tary slackness condition.
112
6.6. COMPLEMENTARY SLACKNESS CONDITION
You are given the information that x2 and x3 are strictly positive in the optimal solution.
Use the complementary slackness conditions to solve the dual problem.
Since x2 and x3 are positive in the optimal solutions, the corresponding constraints in
the dual problem become equalities. Thus y = (y1 , y2 ) is an optimal solution to the dual
problem if and only if it satisfies the first constraint and the following system
32
y1 + 2y2 = < 5.
7
Note that we can also use this information to find the optimal solution to the primal
problem. In fact, since the first constraint in the dual problem is an inequality, by the
complementary slackness conditions, the first variable in the primal problem is zero, that
is x1 = 0. Substituting this to the primal constraints, we obtain x2 = 22
7
, x3 = 26
7
, which
are nonnegative. Also we can verify that the objective values coincide:
368
10y1 + 8y2 = 5x1 + 12x2 + 4x3 = .
7
Thus y = (40/7, −4/7) and x = (0, 22/7, 26/7) are respectively optimal solutions to the
primal and the dual problems.
113
6.7. MORE APPLICATIONS OF DUALITY THEORY
Theorem 6.25 (Strict Complementarity). If primal and dual problems both have feasible
solutions, then both problems have a pair of strictly complementary solutions x∗ ≥ 0 and
s∗ ≥ 0 with
(x∗ )T s∗ = 0, x∗ + s∗ > 0.
The proof of this theorem can be found in [Gre94] where it is proved that a primal-
dual pair of optimal solutions is unique if and only if it is a strictly complementary pair
of basic solutions, see also Remark 5 in Chapter 8 for more discussions.
Minimize x1 + x2 + 6x3
s.t. −x1 + x2 + x3 ≥ 2,
x1 − 2x2 + 2x3 ≥ 6,
x1 , x2 , x3 ≥ 0.
Example 6.27. Using duality theory to prove the Farkas’s Lemma: Only one of the
following systems has a solution (feasible)
• (System 1) Ax = b, x ≥ 0.
• (System 2) AT y ≤ 0, bT y > 0.
Proof. If System 1 has a solution, then for any y such that AT y ≤ 0 we have
bT y = (Ax)T y = xT (AT y) ≤ 0.
114
6.7. MORE APPLICATIONS OF DUALITY THEORY
The inequality follows from the fact that AT y ≤ 0 and x ≥ 0. So, the System 2 is
impossible to have a solution.
Minimize 0T x
s.t. Ax = b, x ≥ 0
is infeasible. According to the duality theorem, its dual problem which is given as
Maximize bT y
s.t. AT y ≤ 0
115
Chapter 7
[BT97, Chapter 3] and [DT97, Chapter 3] contain relevant material to this chapter that
you may find useful.
116
7.1. BASIC FEASIBLE SOLUTION
For matrices, we may perform some elementary row operations. These operations are
most helpful in solving a system of linear equations and in finding the inverse of a matrix.
By row operations, A0 can be upper triangular, and then we can solve the system by
back substitution. The process of reducing A to an upper triangular matrix with ones on
the diagonal is called Gaussian Reduction of the system, by further reduction, we may
reduce A to the identity matrix, the process is called a Gauss-Jordan reduction of the
system.
min{cT x : Ax = b, x ≥ 0},
117
7.1. BASIC FEASIBLE SOLUTION
Theorem 7.2 (Existence of Optimal Extreme Point). Assume that the feasible region is
nonempty. Then
• Otherwise, if there is an extreme direction such that cT dj < 0, then the optimal
objective value of LP is unbounded.
Proof. Let x1 , x2 , ..., xk be the extreme points of the constraint set. And let d1 , ..., dl be
the extreme directions. By Minkoski’s theorem, any feasible point x can be represented
as
k
X l
X
j
x= λj x + µj dj ,
j=1 j=1
where
k
X
λj = 1, λj ≥ 0, j = 1, ..., k,
j=1
µj ≥ 0, j = 1, ..., l.
Since the µj ’s can be made arbitrarily large, the minimum is −∞ if cT dj < 0 for some j.
Thus, the necessary condition for the existence of the optimal solution is
cT dj ≥ 0, ∀j = 1, ..., l.
If this condition holds, we can show that the optimal solution exists and attains at an
extreme point. In fact, if cT dj ≥ 0 for j ≤ l, in order to minimize the objective function,
µj should be taken to be zero for all j. In order to minimize the first term of the objective,
we simply find the minimum cT xj , say cT xp , and let λp = 1, and all other λj ’s equal to
zero.
118
7.1. BASIC FEASIBLE SOLUTION
• In summary, the optimal solution of the linear programming is finite if and only if
cT dj ≥ 0 for all extreme directions.
• Furthermore, if this is the case, then we find the minimum point by picking the
minimum objective value among all extreme points.
• This shows that if an optimal solution exists, we must be able to find an optimal
extreme point.
• If the current extreme point is not optimal, how to move from it to the next better
extreme point?
To address the first issue, we need the concept of Basic Feasible Solution (or basic
feasible point).
Ax = b, x ≥ 0.
A = [B, N ],
where
B is an m × m invertible matrix and N is an m × (n − m) matrix. Denote
xB
x= . Then the system become
xN
xB
[B, N ] = b,
xN
119
7.1. BASIC FEASIBLE SOLUTION
i.e.,
BxB + N xN = b.
Note that
xB = B −1 b, xN = 0
is a solution to the above system. This solution is called a basic solution. Thus a basic
solution has at least n−m zero variables. In addition, if x is a basis solution then columns
of A corresponding to non-zero variables of x are linearly independent.
• xB —– basic variable
• xN —– nonbasic variable
x1 + x2 + x3 = 6,
x2 + x4 = 3,
x1 , x2 , x3 , x4 ≥ 0.
120
7.1. BASIC FEASIBLE SOLUTION
1 0 x1 6 x2 0
2. B = [a1 , a4 ] = , xB = = B −1 b = , xN = = .
0 1 x4 3 x3 0
1 1 x2 3 x1 0
3. B = [a2 , a3 ] = , xB = = B −1 b = , xN = = .
1 0 x3 3 x4 0
1 0 x2 6 x1
4. B = [a2 , a4 ] = , xB = = B −1 b = , xN = =
1 1 x4 −3 x3
0
.
0
1 0 x3 6 x1 0
5. B = [a3 , a4 ] = , xB = = B −1 b = , xN = = .
0 1 x4 3 x2 0
Note that the points corresponding to 1,2,3, and 5 are basic feasible solutions. The
point obtained in 4 is not a basic feasible solution because it violates the nonnegativity
restrictions. In other words, we have four basic feasible solutions, namely
3 6 0 0
3 0 3 0
x1 =
, x2 = , x3 = , x4 = .
0 0 3 6
0 3 0 3
These points belong to R4 since after introducing the slack variables we have four variables.
There basic feasible solutions, projected in R2 —that is, in the (x1 , x2 )-plane — give rise
to the following four points (3, 3), (6, 0), (0, 3) and (0, 0). These points are precisely the
extreme points of the feasible region.
The possible number of basic feasible solutions is bounded by the number of ways of
extracting m columns from A to form the basis, so in general,
n n!
the number of basic feasible solution ≤ = .
m m!(n − m)!
121
7.1. BASIC FEASIBLE SOLUTION
Theorem 7.5. A point is a basic feasible solution if and only if it is an extreme point.
Ay = ByB + N yN = b = Az = BzB + N xN
1 1
x = (x + y) + (x − y).
2 2
Thus, the set of basic feasible solutions coincides with the set of extreme points (the
former is an algebraic description and the latter is a geometric description). Since an LP
with a finite optimal value has an optimal solution at an extreme point, an optimal basic
feasible solution can always be found for such problems.
122
7.1. BASIC FEASIBLE SOLUTION
ii) If there is an optimal solution for LP, there is a basic feasible solution that is
optimal.
a1 x1 + . . . ap xp = b, (7.1)
Case 2: If {a1 , . . . , ap } are linearly dependent. Then there exist y1 , . . . , yp , not all zero,
such that
y1 a1 + . . . + yp ap = 0. (7.2)
Without loss of generality we can assume that yi > 0 for some i ∈ {1, . . . , p} (otherwise,
we can multiply the above equation by −1). From (1.6) and (7.2) we deduce that
Hence A(x − εy) = b where y = (y1 , . . . , yp , 0, . . . , 0)T . For sufficiently small ε we have
We choose ε by
nx o
ε = min
i
i = 1, . . . , p; yi > 0 .
yi
Then we have
(x − εy) ≥ 0 and A(x − εy) = b.
Thus (x − εy) is a feasible solution. In addition, due to the choice of ε, the first p
coordinates of (x − εy) are non-negative and at least one of them is zero. Hence we reduce
123
7.1. BASIC FEASIBLE SOLUTION
Part ii). Suppose that x is an optimal solution. Without loss of generality we assume
that x has minimal number of positive ordinates. We will construct a basic feasible
optimal solution from x. If x = 0 then x is basic (and the cost is zero). If x 6= 0, let
I = {i1 , . . . , ip } is the set of index such that xij > 0. Similarly as in Part i), there are two
cases
Case 1: the column vectors {ai1 , . . . , aip } of A are linearly independent. This case we can
proceed exactly the same as in Part i) and obtain a basic feasible optimal solution. Note
that the optimality does not change in this case.
Case 2: the column vectors {ai1 , . . . , aip } of A are linearly dependent. We also proceed
similarly as in the second case in Part i. However, we need to show that the optimality
is preserved during the process. To show that x − εy is optimal, we consider its objective
value
cT (x − εy) = cT x − εcT y
For sufficiently small |ε|, the vector x = εy is a feasible solution (for positive or negative
values of ε. It follows that we must have cT y = 0, since otherwise, we could find a small
ε of the proper sign such that cT (x − εy) < cT x, which would violate the optimality of x.
Thus x − εy is still optimal.
Let us see how to construct a BFS from the feasible point x = (3, 2, 1, 1, 1)T . The given x
has no zero entries, so if we try to replicate the proof of the theorem 7.6, we have m = 3
and p = 5. Therefore we need to solve the system Ay = 0 and pick one solution, for
example y = (0, 0, 1, −1, −1)T .
n
Note that y has only one component y3 that is strictly positive, so ε = min xyii i =
o
1, . . . , p; yi > 0 = xy33 = 1 and x − y = (3, 2, 0, 2, 2)T . We take this as our new x̃ =
124
7.2. THE SIMPLEX METHOD
(3, 2, 0, 2, 2)T this is still a feasible point. It has one coordinate equal to 0 and therefore
Ax̃ = Ãx̃ with
−1 3 0 0
à = 0 2 −1 0 .
0 0 −1 1
Let us solve the system Ãỹ = 0 and pick one solution, for example (3, 1, 2, 2)T , namely
n o n o
ỹ = (3, 1, 2, 2)T . We calculate ε = min x̃ỹii i = 1, . . . , 5; yi > 0 = min 33 , 21 , 22 , 22 = 1..
Again we take x̃ − ỹ = (0, 1, 0, 0, 0)T . This is a BFS. Infact, if we selct the matrix B made
of the second, fourth and fifth columns of A, we can see that x̃ − ỹ = B −1 b.
In the next session, we are going to study how to solve LPs by the simplex method.
• The simplex method is to proceed from one BFS to an adjacent or neighboring one
in such a way as to continuously improve the value of the objective function (moving
from an extreme point to another with a better at least not worse objective).
• It also discovers whether the feasible region is empty, and whether the optimal
solution is unbounded. In practice, the method only enumerates a small portion of
the extreme points of the feasible region.
• The key to the simplex method lies in recognizing the optimality of a given extreme
point solution based on local consideration without having to enumerate all extreme
points or basic feasible solutions.
• The basic idea of the simplex method is to confine the search to extreme points of
the feasible region in a most intelligent way. The key for the simplex method is to
make computers find extreme points.
125
7.2. THE SIMPLEX METHOD
For simplicity, throughout the remainder of this course, the vector c is understood as
a row vector. We always assume that the problem is in standard form
min cx
s.t. Ax = b, x ≥ 0,
where A is an m × n matrix with full rank, i.e., rank(A) = m, and b a vector in Rm , and
c is a vector in Rn .
For this given basis B, we would like to check if it is an optimal solution of the
problem.
If it is not, we would like to find another basic feasible solution at which the objective
value is smaller than the current objective value f0 .
Let x be a feasible solution of Ax = b, and let xB and xN be its basic and nonbasic
variables for this given basis. Thus, we have
xB
b = Ax = [B, N ] = BxB + N xN .
xN
Multiplying by B −1 we have
X X
xB = B −1 b − B −1 N xN = B −1 b − B −1 aj xj = b̄ −
y j xj
j∈R j∈R
where R denotes the current set of indices of the nonbasic variables, and
b̄ = B −1 b
126
7.2. THE SIMPLEX METHOD
yj = B −1 aj , j ∈ R.
f = cx
= cB xB + cN xN
= (cB B −1 b − cB B −1 N xN ) + cN xN
= cB B −1 b − cB B −1 N − cN xN
= f0 − (z − cN )xN
X
= f0 − (zj − cj )xj ,
j∈R
i.e.,
zj = cB B −1 aj = cB yj , j ∈ R.
Minimize
X
f = f0 − (zj − cj )xj
j∈R
subject to (7.3)
X
(yj )xj + xB = b̄
j∈R
xj ≥ 0, j ∈ R, and xB ≥ 0.
For this problem, xB can be viewed as a slack variable, so the above LP can be further
rewritten as
Minimize
X
f = f0 − (zj − cj )xj
j∈R
subject to (7.4)
X
(yj )xj ≤ b̄
j∈R
xj ≥ 0, j ∈ R.
Definition:
127
7.2. THE SIMPLEX METHOD
Proposition 7.8. From (7.4), we see that if (zj − cj ) ≤ 0 for all j ∈ R, then the current
basic feasible solution is optimal.
for any feasible solution. For the current (basic) feasible solution, we know that f = f0
since xj = 0 for all j ∈ R. So, the current feasible solution is optimal.
then LP has an unbounded solution (or simply, we say that the LP is unbounded)
since we can send xj to +∞ in (7.4).
has a positive component. In this case, the current objective value can be further
decreased (if the current BFS is non-degenerated).
The simplex procedure moves from one basis to an adjacent basis, by introducing one
variable into the basis, and removing another variable from the basis. The criteria for
entering and leaving are summarized below.
128
7.2. THE SIMPLEX METHOD
• Leaving: If
b̄r b̄i
≡ min : yik > 0
yrk 1≤i≤m yik
then xBr (the rth component of xB ) may leave the basis.
3. Otherwise, according to the above rule to choose entering variable, and leaving variable,
to form new basis, and get improved basic feasible solution.
7.2.3 Analysis for the case zj −cj > 0 and yj = B −1 aj has a positive
component for some nonbasic variable
In fact, choose one of the positive number like zk − ck , and perhaps the most positive
of all the zj − cj , j ∈ R. Set
xj = 0, for j ∈ R − {k}.
Denote
y
1k
y2k
..
.
yk =
.
yrk
..
.
ymk
We see from (7.3) that
f = f0 − (zk − ck )xk (7.5)
129
7.2. THE SIMPLEX METHOD
and
x b̄ y1k
B1 1
xB2 b̄2 y2k
.. .. ..
. . .
(7.6)
xB = = b̄ − (yk )xk = − xk .
xBr b̄r yrk
.. .. ..
. . .
xBm b̄m ymk
Assume the current basic feasible solution is not degenerated, i.e. b̄ > 0. Now let us
analyze what will happen when we increase the value of xk .
(ii) For those yik ≤ 0, when xk increases, (7.6) implies that the component xBi will
increase, continuing to be positive.
(iii) For those yik > 0, increasing xk , the corresponding component xBi will decrease.
From (7.6), the first basic variable dropping to zero corresponds to the minimum of b̄i /yik
for positive yik . More precisely, we can increase xk until
b̄r b̄i
xk = ≡ min : yik > 0 . (7.7)
yrk 1≤i≤m yik
Under non-degeneracy, xk > 0. From (7.5) and fact that zk − ck > 0, it follows that
objective can be strictly improved, As xk increases from level 0 to br /yrk , a new improved
feasible solution is obtained. Substituting xk given by (7.7) into (7.6) yields the following:
x = b̄i − yyrkik
b̄r , i = 1, 2, ..., m,
Bi
xk = yb̄rkr , (7.8)
all other (x )0 s are zero.
j
From (7.8), we have xBr = 0, and hence at most m variables are positive. The
corresponding columns in A are
130
7.2. THE SIMPLEX METHOD
Since yrk 6= 0, when the rth column of B is replaced by ak , the resulting vectors are still
linearly independent. This fact follows from the next Lemma, which answers the following
natural question: If we have a basis of Rn , what is the condition that will guarantee that
if a vector of the basis, say aj , is replaced by an another vector, say, a, then the new set
of vectors still forms a basis?
Lemma 7.10. Let a1 , a2 , ..., an form a basis of Rn . Then any vector a can be represented
as
n
X
a= λ j aj .
j=1
Let’s replace aj by a, then the vectors a1 , ..., aj−1 , a, aj+1 , ..., an are linearly independent if
and only if, in the above representation, λj 6= 0.
Proposition 7.11. (7.8) is the a basic feasible solution corresponding the basis
B
e = [aB1 , aB2 , ..., aBr−1 , ak , aBr+1 , ..., aBm ],
131
7.2. THE SIMPLEX METHOD
where xB1 , xB2 , ..., xk , ..., xBm is given by (7.8). In fact, the left-hand-side
y1k b̄r ymk
b̄1 − b̄r aB1 + ... + ak + ... + b̄m − b̄r aBm
yrk yrk yrk
= b̄1 − y1k xk aB1 + ... + xk ak + ... + b̄m − ymk xk aBm
X m
= b̄j aBj + −b̄r aBr − y1k xk aB1 − ... − y(r−1)k xk aBr−1 + xk ak − ... − ymk xk aBm
j=1
m m
!
X X
= b̄j aBj + xk − yjk aBj + ak
j=1 j=1
= B b̄ + xk (ak − B(yk ))
=b+0=b
132
Chapter 8
In this chapter, we continue studying the simplex method focusing on the simplex
algorithm algorithmic aspect. We will study how to implement the simplex method in
practices. The following topics will be covered
[BT97, Chapter 3] and [DT97, Chapter 3] have similar material in this chapter that you
may find useful.
Initial Step
133
8.1. THE ALGORITHM FOR MINIMIZATION PROBLEMS
Main Step
xB = B −1 b = b̄,
xN = 0,
f = cB xB = cB B −1 b = cB b̄.
Set
zk − ck = max{zj − cj },
j∈R
where R is the current set of indices associated with the nonbasic variables.
Step 4. Let xk enter the basis. The index r of the current basic variable xBr which
leaves the basis is determined by the following minimum ratio test:
b̄r b̄i
≡ min : yik > 0
yrk 1≤i≤m yik
Update the basis B where ak replaces aBr , update the index set R, and go to step 1.
134
8.2. THE SIMPLEX METHOD IN TABLEAU FORMAT
• or else we generate a new basic feasible solution if zk − ck > 0 and yk 6≤ 0, and the
objective function strictly decreases.
Therefore, in the absence of degeneracy (i.e. xB > 0), the objective strictly decreases
at each iteration, and hence the basic feasible solution generated by the simplex method
are distinct. Since there is only a finite number of basic feasible solutions, the method
would stop in a finite number of steps with either an optimal solution or an unbounded
optimal value.
Minimize f
subject to f − cB xB − cN xN = 0 (8.1)
BxB + N xN = b (8.2)
xB , xN ≥ 0.
cB xB + cB B −1 N xN = cB B −1 b.
f + 0xB + cB B −1 N − cN xN = cB B −1 b.
135
8.2. THE SIMPLEX METHOD IN TABLEAU FORMAT
So the LP is reformulated as
Minimize f
subject to f + 0xB + cB B −1 N − cN xN = cB B −1 b,
xB + B −1 N xN = B −1 b,
xB , xN ≥ 0.
xB xN RHS
f 0 cB B −1 N − cN cB B −1 b
xB I B −1 N B −1 b
The table contains all information of the linear programming that we need to proceed
with the simplex method:
136
8.2. THE SIMPLEX METHOD IN TABLEAU FORMAT
8.2.1 Pivoting
If xk enters the basis and xBr leaves the basis, then pivoting on yrk includes the
following operations:
• For i = 1, ..., m and i 6= r, update the ith row by adding to it −yik times the new
rth row.
• Update row zero by adding to it −(zk − ck ) times the new rth row.
The following two tables show the situation before and after pivoting operations.
xB 1 xB r xB m xj xk RHS
• xk entered and xBr left the basis. This is illustrated on the left-hand-side of the
tableau by replacing xBr with xk .
137
8.2. THE SIMPLEX METHOD IN TABLEAU FORMAT
xB1 xB r xB m xj xk RHS
(zj − cj )
f 0 ··· ck −ck
yrk
··· 0 ··· ··· 0 ··· cB b̄ − (zk − ck ) yb̄rkr
yrj
− yrk (zk − ck )
1 0 0
−y1k yrj y1k
xB 1 ··· yrk
··· ··· y1j − yrk y1k ··· ··· b̄1 − b̄
yrk r
.. .. .. .. .. .. ..
. . . . . . .
0 0 1
1 yrj b̄r
xk ··· yrk
··· ··· yrk
··· ··· yrk
.. .. .. .. .. .. ..
. . . . . . .
0 1 0
−ymk yrj ymk
xBm ··· yrk
··· ··· ymj − y
yrk mk
··· ··· b̄m − b̄
yrk r
• The right-hand-side gives the current values of the basic variables. The nonbasic
variables are kept zero.
• Suppose that the original columns for the new basic and nonbasic variables are B̂
and N̂ , respectively. The pivoting operation which is a sequence of elementary row
operations results in a new tableau that gives the new B̂ −1 N̂ under the nonbasic
variables, and updated set of zj − cj ’s for the new nonbasic variables, and the values
of the new basic variables and objective.
Initialization step:
Find an initial basic feasible solution with basis B. Form the following initial tableau.
xB xN RHS
f 0 cB B −1 N − cN cB B −1 b
xB I B −1 N B −1 b
Main Step:
138
8.2. THE SIMPLEX METHOD IN TABLEAU FORMAT
Step 2. Examine yk .
• Otherwise, go to Step 3
Step 3. Determine the index r by minimum ratio test to find the variable xBr to
leave.
Step 4.
• Update the basic and nonbasic variables where xk enters the basis and xBr leaves
the basis, and repeat the main step.
Minimise x1 + x2 − 4x3
s.t. x1 + x2 + 2x3 ≤ 9,
x1 + x2 − x3 ≤ 2,
−x1 + x2 + x3 ≤ 4,
x1 , x2 , x3 ≥ 0.
subject to x1 + x2 + 2x3 + x4 = 9,
x1 + x2 − x3 + x5 = 2,
−x1 + x2 + x3 + x6 = 4,
x1 , x2 , x3 , x4 , x5 , x6 ≥ 0.
Initial iteration
139
8.2. THE SIMPLEX METHOD IN TABLEAU FORMAT
x1 x2 x3 x4 x5 x6 RHS
f −1 −1 4 0 0 0 0
x4 1 1 2 1 0 0 9
x5 1 1 −1 0 1 0 2
x6 −1 1 1∗ 0 0 1 4
• From this table, we see that x3 is a nonbasic variable, and z3 − c3 = 4 > 0. Thus,
the current basic feasible solution (x4 , x5 , x6 ) = (9, 2, 4) is not optimal. x3 will enter
the basis.
• Now we are going to determine which vector among x4 , x5 , x6 will leave the basis.
By checking y3 = (2, −1, 1)T , we note that it has two positive components. By the
minimum ratio test: min{ 92 , 41 } = 4. The vector x6 leaves the basis, and the marked
elements in the above table is the pivoting element.
• We do pivoting operations to get a new basic feasible solution. Thus the algorithm
proceeds to the next iteration.
Iteration 1.
x1 x2 x3 x4 x5 x6 RHS
f 3 −5 0 0 0 −4 −16
x4 3∗ −1 0 1 0 −2 1
x5 0 2 0 0 1 1 6
x3 −1 1 1 0 0 1 4
• Now we determine which vector among x4 , x5 , x3 will leave the basis. By checking
y1 = (3, 0, −1)T and the minimum ratio test indicates that x4 should leave the basis.
Iteration 2.
140
8.3. DEGENERACY AND CYCLING IN THE SIMPLEX METHOD
x1 x2 x3 x4 x5 x6 RHS
f 0 −4 0 −1 0 −2 −17
x1 1 −1/3 0 1/3 0 −2/3 1/3
x5 0 2 0 0 1 1 6
x3 0 2/3 1 1/3 0 1/3 13/3
Now, zj − cj ≤ 0 for all nonbasic variables, and thus an optimal solution is found and
it is given by
x∗1 = 1/3, x∗2 = 0, x∗3 = 13/3.
141
8.3. DEGENERACY AND CYCLING IN THE SIMPLEX METHOD
Example 8.3 (Beale 1955). Solve the following LP by using simplex method:
3 1
Minimise − x1 + 150x2 − x3 + 6x4
4 50
1 1
subject to x1 − 60x2 − x3 + 9x4 ≤ 0,
4 25
1 1
x1 − 90x2 − x3 + 3x4 ≤ 0,
2 50
x3 ≤ 1,
x1 , x2 , x3 , x4 ≥ 0.
Proof. In each iteration below, the pivoting entry is marked by an asterisk. Adding slack
variables x5 , x6 , x7 to obtain the following problem
3 1
Minimise − x1 + 150x2 − x3 + 6x4
4 50
1 1
subject to x1 − 60x2 − x3 + 9x4 + x5 = 0,
4 25
1 1
x1 − 90x2 − x3 + 3x4 + x6 = 0,
2 50
x3 + x7 = 1,
x1 , x2 , x3 , x4 , x5 , x6 , x7 ≥ 0.
Iteration 1.
x1 x2 x3 x4 x5 x6 x7 RHS
f 3/4∗ −150 1/50 -6 0 0 0 0
x5 1/4 -60 -1/25 9 1 0 0 0
x6 1/2 -90 -1/50 3 0 1 0 0
x7 0 0 1 0 0 0 1 1
Iteration 2.
x1 x2 x3 x4 x5 x6 x7 RHS
f 0 30 7/50 -33 -3 0 0 0
x1 1 -240 -4/25 36 4 0 0 0
x6 0 30∗ 3/50 -15 -2 1 0 0
x7 0 0 1 0 0 0 1 1
Iteration 3.
142
8.3. DEGENERACY AND CYCLING IN THE SIMPLEX METHOD
x1 x2 x3 x4 x5 x6 x7 RHS
f 0 0 2/25 -18 -1 -1 0 0
x1 1 0 8/25∗ -84 -12 8 0 0
x2 0 1 1/500 -1/2 -1/15 1/30 0 0
x7 0 0 1 0 0 0 1 1
Iteration 4.
x1 x2 x3 x4 x5 x6 x7 RHS
f -1/4 0 0 3 2 -3 0 0
x3 25/8 0 1 -525/2 -75/2 25 0 0
x2 -1/60 1 0 1/40∗ 1/120 -1/60 0 0
x7 -25/8 0 0 525/2 75/2 -25 1 1
Iteration 5.
x1 x2 x3 x4 x5 x6 x7 RHS
f 1/2 -120 0 0 1 -1 0 0
x3 -125/2 10500 1 0 50∗ -150 0 0
x4 -1/4 40 0 1 1/3 -2/3 0 0
x7 125/2 10500 0 0 50 150 1 1
Iteration 6.
x1 x2 x3 x4 x5 x6 x7 RHS
f 7/4 -330 -1/50 0 0 2 0 0
x5 -5/4 210 1/50 0 1 -3 0 0
x4 1/6 -30 -1/150 1 0 1/3∗ 0 0
x7 0 0 1 0 0 0 1 1
Iteration 7.
x1 x2 x3 x4 x5 x6 x7 RHS
f 3/4∗ −150 1/50 -6 0 0 0 0
x5 1/4 -60 -1/25 9 1 0 0 0
x6 1/2 -90 -1/50 3 0 1 0 0
x7 0 0 1 0 0 0 1 1
143
8.4. EFFICIENCY OF THE SIMPLEX METHOD
This tableau is exactly as the initial tableau. Therefore, the simplex method will cycle
endlessly. We observe that the basic feasible solution in each iteration is degenerate
(but not totally degenerate) and the objective function is actually does not change. It is
this degeneracy that causes cycling. There are several ways to avoid cycling. The most
commonly used method is Bland’s rule which is formally stated as follows:
1. Choose the lowest-numbered (i.e., leftmost) nonbasic column with a negative (re-
duced) cost.
2. If there are more that one row that satisfy the minimum ratio test, choose the row
with the lowest-numbered column (variable) basic in it.
Let us apply the Bland’s rule to the above example. In the Iteration 5. we choose x1 to
enter instead of x5 . Iteration 5’.
x1 x2 x3 x4 x5 x6 x7 RHS
f 1/2 -120 0 0 1 -1 0 0
x3 -125/2 10500 1 0 50 -150 0 0
x4 -1/4 40 0 1 1/3 -2/3 0 0
x7 125/2∗ 10500 0 0 50 150 1 1
144
8.4. EFFICIENCY OF THE SIMPLEX METHOD
subject to x1 ≤ 5,
4x1 + x2 ≤ 25,
2n x1 + 2n−1 x2 + . . . + 4xn−1 + xn ≤ 5n ,
x1 , . . . , xn ≥ 0.
The LP has n variables, n constraints and 2n extreme points. The simplex method,
starting from the origin, will go through each of the extreme points before obtaining the
optimal solution at (0, 0, . . . , 5n ). To illustrate this, we consider the Klee-Minty cube for
the case n = 3, that is
subject to x1 ≤ 5,
4x1 + x2 ≤ 25,
x1 , x2 , x3 ≥ 0.
It is easy to see that 4x1 + 2x2 + x3 ≤ 8x1 + 4x2 + x3 = 125 and the inequality becomes
equality when x1 = x2 = 0, x3 = 125. Thus the optimal value is 125 and is achieved
at (x1 , x2 , x3 ) = (0, 0, 125). Adding s1 , s2 , s3 as slack variables to obtain the following
problem
subject to x1 + s1 = 5,
4x1 + x2 + s2 = 25,
x1 , x2 , x3 , s1 , s2 , s3 ≥ 0.
145
8.4. EFFICIENCY OF THE SIMPLEX METHOD
x1 x2 x3 s1 s2 s3 RHS x1 x2 x3 s1 s2 s3 RHS
−f 4 2 1 0 0 0 0 −f 0 2 1 -4 0 0 -20
s1 1 0 0 1 0 0 0 x1 1 0 0 1 0 0 5
s2 4 1 0 0 0 0 0 s2 0 1 0 -4 1 0 5
s3 8 4 1 0 0 1 0 s3 0 4 1 -8 0 1 85
x1 x2 x3 s1 s2 s3 RHS x1 x2 x3 s1 s2 s3 RHS
−f 0 0 1 4 -2 0 -30 −f -4 0 1 0 -2 0 -50
x1 1 0 0 1 0 0 5 s1 0 0 0 1 0 0 5
x2 0 1 0 -4 1 0 5 x2 4 1 0 0 1 0 25
s3 0 0 1 8 -4 1 65 s3 -8 0 1 0 -4 1 25
x1 x2 x3 s1 s2 s3 RHS x1 x2 x3 s1 s2 s3 RHS
−f 4 0 0 0 2 -1 -75 −f 0 0 0 -4 2 -1 -95
s1 1 0 0 1 0 0 5 x1 1 0 0 1 0 0 5
x2 4 1 0 0 1 0 25 x2 0 1 0 -4 1 0 5
x3 -8 0 1 0 -4 1 25 x3 0 0 1 8 -4 1 65
x1 x2 x3 s1 s2 s3 RHS x1 x2 x3 s1 s2 s3 RHS
−f 0 -2 0 4 0 -1 -105 −f -4 -2 0 0 0 -1 -125
x1 1 0 0 1 0 0 5 s1 1 0 0 1 0 0 5
s2 0 1 0 -4 1 0 5 s2 4 1 0 0 1 0 25
x3 0 4 1 -8 0 1 85 x3 8 4 1 0 0 1 125
Thus the simplex method applied to the Klee-Minty cube for n = 3, starting at the origin,
goes through all of the 8 vertices before reaching the optimal value.
146
8.4. EFFICIENCY OF THE SIMPLEX METHOD
Minimise x1 + x2 + 2x3
subject to x1 + x3 = 1,
x2 + x3 = 1,
x3 ≤ 0,
x1 , x2 ≥ 0.
This problem has multiple solution, for instance (1, 1, 0) and (0, 1, 1). In fact, any feasible
point is optimal. The dual problem is
Maximise y1 + y2
subject to y1 ≤ 1,
y2 ≥ 1,
y1 + y2 ≤ 2.
The dual also has multiple solutions, for example (1, 1) and (0, 2).
Remark 6. There are other algorithms for solving linear-programming problems. In par-
ticular, interior-point methods refer to class of methods for solving linear programming
problems that move through the interior of the feasible region such as Khachiyan’s ellip-
soidal algorithm and Karmarkar’s algorithm. This is in contrast to the simplex algorithm,
which finds an optimal solution by traversing the edges between vertices of the feasible
region. Interior-point methods often have worst-case polynomial complexity; however, in
most practical applications, the simplex method is often very efficient.
The linear programming problem: Find a strongly-polynomial time algorithm which for
given matrix A ∈ Rm×n and b ∈ Rm decides whether there exists x ∈ Rn with Ax ≥ b.
147
8.5. FURTHER EXAMPLES FOR SIMPLEX METHODS & DEGENERACY
Minimise −3x1 + x2
s.t. x1 + 2x2 ≤ 4,
−x1 + x2 ≤ 1,
x1 , x2 , x3 , x4 ≥ 0.
s.t. x1 − 2x2 ≤ 4,
−x1 + x2 ≤ 3,
x1 , x2 ≥ 0.
x1 x2 x3 x4 RHS
f 1 3 0 0 0
x3 1 −2 1 0 4
x4 −1 1∗ 0 1 3
x1 x2 x3 x4 RHS
f 4 0 0 −3 −9
x3 −1 0 1 2 10
x2 −1 1 0 1 3
148
8.5. FURTHER EXAMPLES FOR SIMPLEX METHODS & DEGENERACY
Maximise −3x1 + x2
2x1 + x2 ≥ −2,
2x1 + x2 ≤ 8,
Degeneracy cases
When the right-hand-side vector b̄ is not positive (at least one of its components is
zero), we can still use the simplex method to solve the problem. The following is such an
example.
Example 8.8 (Investment problem). Someone has $ 20,000 to finance various invest-
ments. There are five categories of investments, each with an associated return and risk:
His goal is to allocate the money to the categories so as to maximize the return, subject
to the following constraints:
(a) An average risk is no more than 6 (all averages taken over the invested money, not
over the savings).
b) The amount in mortgages and personal loans combined should be no higher than the
amount in commercial.
149
8.5. FURTHER EXAMPLES FOR SIMPLEX METHODS & DEGENERACY
• x3 and x4 are the amounts invested in personal loans and savings, respectively.
subject to x1 − x2 + x3 ≤ 0,
−4x1 + 2x3 ≤ 0,
x1 + x2 + x3 + x4 = 20, 000,
x1 , x2 , x3 , x4 ≥ 0.
subject to x1 − x2 + x3 + x5 = 0,
−4x1 + 2x3 + x6 = 0,
x1 + x2 + x3 + x4 = 20, 000,
x1 , x2 , x3 , x4 , x5 , x6 ≥ 0.
x1 x2 x3 x4 x5 x6 RHS
f 8 10 12 3 0 0 0
x5 1 −1 1 0 1 0 0
x6 −4 0 2 0 0 1 0
x4 1 1 1 1 0 0 20,000
150
8.5. FURTHER EXAMPLES FOR SIMPLEX METHODS & DEGENERACY
x1 x2 x3 x4 x5 x6 RHS
f 5 7 9 0 0 0 −60, 000
x5 1 −1 1∗ 0 1 0 0
x6 −4 0 2 0 0 1 0
x4 1 1 1 1 0 0 20,000
x1 x2 x3 x4 x5 x6 RHS
f −4 16 0 0 −9 0 −60, 000
x3 1 −1 1 0 1 0 0
x6 −6 2∗ 0 0 −2 1 0
x4 0 2 0 1 −1 0 20,000
Replacing x6 by x2 , we have
x1 x2 x3 x4 x5 x6 RHS
f 44 0 0 0 7 −8 −60, 000
x3 −2 0 1 0 0 1/2 0
x2 −3 1 0 0 −1 1/2 0
x4 6∗ 0 0 1 1 −1 20,000
Replacing x4 by x1 , we have
x1 x2 x3 x4 x5 x6 RHS
f 0 0 0 −22/3 −1/3 −2/3 −620, 000/3
x3 0 0 1 1/3 1/3 1/6 20,000/3
x2 0 1 0 1/2 −1/2 0 10,000
x1 1 0 0 1/6 1/6 −1/6 10,000/3
151
8.6. EXTRACTING INFORMATION FROM OPTIMAL SIMPLEX TABLEAU
(P1 ) is −620, 000/3, and thus the maximum return for the original problem (P2 ) is
620, 000
f∗ = .
3
Suppose that we get the final tableau (optimal tableau) of simplex method. What
information can be obtained from this final tableau? and how this information can be
used?
x1 x2 x3 x4 x5 x6 RHS
f 0 0 0 −22/3 −1/3 −2/3 −620, 000/3
x3 0 0 1 1/3 1/3 1/6 20,000/3
x2 0 1 0 1/2 −1/2 0 10,000
x1 1 0 0 1/6 1/6 −1/6 10,000/3
f ∗ = −620, 000/3.
152
8.6. EXTRACTING INFORMATION FROM OPTIMAL SIMPLEX TABLEAU
w∗ = cB B −1 ,
where B is the optimal basis. For the following two cases, we may find the dual optimal
solution immediately without any effort from the optimal tableau.
min{cx : Ax ≥ b, x ≥ 0}.
which are given by the “reduced cost coefficient” of slack variables xn+i (i = 1, ..., m) in
the optimal Tableau.
min{cx : Ax ≤ b, x ≥ 0},
which are “reduced cost coefficients” for slack variables xn+i (i = 1, ..., m) in the optimal
tableau.
to determine the dual optimal solution, we need to know B −1 . Is this information available
in optimal tableau?
153
8.6. EXTRACTING INFORMATION FROM OPTIMAL SIMPLEX TABLEAU
In fact, at each iteration, the current B −1 can be obtained immediately from the
current tableau, and the inverse of optimal basis can be found in optimal tableau.
Assume that the original tableau has an identity matrix. The process of reducing the
basis matrix B of the original tableau to an identity matrix in the current tableau, is
equivalent to premultiplying rows 1 through m of the original tableau by B −1 to produce
the current tableau. This also converts the identity matrix of the original tableau to B −1 .
Therefore, B −1 can be extracted from the current tableau as the submatrix in rows 1
through m under the original identity columns.
subject to x1 − x2 + x3 ≤ 0,
−4x1 + 2x3 ≤ 0,
x1 + x2 + x3 + x4 = 20, 000,
x1 , x2 , x3 , x4 ≥ 0.
subject to x1 − x2 + x3 + x5 = 0,
−4x1 + 2x3 + x6 = 0,
x1 + x2 + x3 + x 4 = 20, 000,
x1 , x2 , x3 , x4 , x5 , x6 ≥ 0.
x1 x2 x3 x4 x5 x6 RHS
f 8 10 12 3 0 0 0
x5 1 −1 1 0 1 0 0
x6 −4 0 2 0 0 1 0
x4 1 1 1 1 0 0 20,000
154
8.6. EXTRACTING INFORMATION FROM OPTIMAL SIMPLEX TABLEAU
x1 x2 x3 x4 x5 x6 RHS
f 0 0 0 −22/3 −1/3 −2/3 −620, 000/3
x3 0 0 1 1/3 1/3 1/6 20,000/3
x2 0 1 0 1/2 −1/2 0 10,000
x1 1 0 0 1/6 1/6 −1/6 10,000/3
What is B and B −1 ?
1 −1 1 1/3 1/6 1/3
B = [a3 , a2 , a1 ] = 2 0 −4 , B −1 = −1/2 1/2 .
0
1 1 1 1/6 −1/6 1/6
Example 8.11. The following simplex tableau shows the optimal solution of a linear
programming. It is known that x4 and x5 are the slack variables in the first and second
constraints of the original problem. The constraints of the original problem are of the ‘≤’
type.
x1 x2 x3 x4 x5 RHS
f 0 −2 0 −3 −2 −35
x3 0 1/4 1 1/2 0 5/2
x1 1 −1/2 0 −1/6 1/3 5/2
The initial identity basis corresponds to x4 and x5 . So in the optimal tableau, the
inverse of the optimal basis B is given by
1/2 0
B −1 = .
−1/6 1/3
In many situations, from the optimal tableau, we may recover the original LP problem.
Example 8.12. (Same example as Example 8.11) The following simplex tableau shows
the optimal solution of a linear programming. It is knows that x4 and x5 are the slack
variables in the first and second constraints of the original problem. The constraints are
of the ‘≤’ type
155
8.6. EXTRACTING INFORMATION FROM OPTIMAL SIMPLEX TABLEAU
x1 x2 x3 x4 x5 RHS
f 0 −2 0 −3 −2 −35
x3 0 1/4 1 1/2 0 5/2
x1 1 −1/2 0 −1/6 1/3 5/2
The data in the simplex tableaux might be lost or corrupted, in which case we need
to recover these data.
Example 8.13. The following is the optimal tableau of simplex method for a given min-
imization problem. The objective is to minimize −2x1 + 3x2 , and the slack variables are
x3 and x4 corresponding to the first and second inequalities, respectively. The constraints
are of the ‘≤’ type.
x1 x2 x3 x4 RHS
f b −1 f g -6
x3 c 0 1 1/5 4
x1 d e 0 2 a
156
Chapter 9
Sensitivity Analysis
• In most practical applications, some of the problem data are uncertain or cannot be
known exactly. In this situation, the problem data are estimated. It is important to
be able to find the new optimal solution of the problem as other estimates of some
of the data become available, without the expensive task of resolving the problem
from scratch.
• Furthermore, in many situations the constraints are not very rigid. For example,
a constraint may reflect the availability of some resources. This availability can
be increased by extra purchase, and the like. It is desirable to examine the effect
of relaxing some of the constraints on the value of the optimal objective without
having to resolve the problem.
These and other related topics constitute sensitivity analysis. [BT97, Chapter 5] and
[DT97, Chapter 7] have similar material in this chapter that you may find useful.
min{cx : Ax = b, x ≥ 0}.
157
9.1. CHANGE IN THE COST VECTOR
We shall describe how to make use of the optimality conditions in order to find a new
optimal solution, if some of the problem data change, without resolving the problem from
scratch. In particular, the following variations in the problem will be considered.
Changes of the cost vector c will change the cost row of the final tableau, that is, the
dual feasibility may be affected (lost). Suppose ck is changed to c0k . This includes two
cases:
We consider only the first case in this course, i.e., ck is changed to c0k and xk is
nonbasic.
• For j 6= k, the reduced cost coefficient zj − cj is not changed, and it satisfies that
zj − cj ≤ 0.
• For j = k, we have
zk − c0k = (zk − ck ) + (ck − c0k ).
Thus, if ck is increased, i.e., if c0k ≥ ck , then we still have zk − c0k ≤ 0, the current
solution remains to be optimal. More general, if c0k ≥ zk , the solution is no change.
Otherwise, zk − c0k > 0, in which case xk must be introduced into the basis and the
simplex method is continued as usual.
158
9.1. CHANGE IN THE COST VECTOR
Minimise −2x1 + x2 − x3
s.t. x1 + x2 + x3 ≤ 6,
−x1 + 2x2 ≤ 4,
x1 , x2 , x3 ≥ 0.
x1 x2 x3 x4 x5 RHS
f 0 −3 −1 −2 0 −12
x1 1 1 1 1 0 6
x5 0 3 1 1 1 10
and all other zj − cj are unaffected. Hence x2 enters the basis to replace x5 in the basis.
So we get the following table
x1 x2 x3 x4 x5 RHS
f 0 1 −1 −2 0 −12
x1 1 1 1 1 0 6
x2 0 3∗ 1 1 1 10
Continue the simplex iteration, and find the new optimal solution as follows.
x1 x2 x3 x4 x5 RHS
f 0 0 −4/3 −7/3 −1/3 −46/3
x1 1 0 2/3 2/3 −1/3 8/3
x2 0 1 1/3 1/3 1/3 10/3
So the solution of the changed LP is (x∗1 , x∗2 ) = (8/3, 10/3), and the optimal value is
−46/3.
159
9.2. CHANGE IN THE RIGHT-HAND-SIDE
• zj − cj = cB B −1 aj − cj has no change,
Minimise −2x1 + x2 − x3
s.t. x1 + x2 + x3 ≤ 6,
−x1 + 2x2 ≤ 4,
x1 , x2 , x3 ≥ 0.
x1 x2 x3 x4 x5 RHS
f 0 −3 −1 −2 0 −12
x1 1 1 1 1 0 6
x5 0 3 1 1 1 10
i) Suppose that the right-hand-side vector b = (6, 4)T is replaced by b0 = (5, 4)T . Check
if this change will affect the current optimal solution of the problem.
Solutions: the initial identity basis corresponds to x4 and x5 . So in the optimal tableau,
the inverse of the optimal basis B is given by
1 0
B −1 = .
1 1
160
9.3. ADDING A NEW ACTIVITY
Suppose that the RHS vector b = (6, 4)T is replaced by b0 = (5, 4)T . We compute
1 0 5 5
B −1 b0 = = > 0.
1 1 4 9
Thus the current basis is still optimal. The optimal solution become
x1 5
xB = = B −1 b0 = .
x5 9
Now suppose that the RHS vector is changed to b0 = (−3, 4)T . We compute
1 0 −3 −3
B −1 b0 = = .
1 1 4 1
Since B −1 b0 has a negative entry, the current basis becomes infeasible. The dual simplex
method (which will be studied in Chapter 11) implies that the dual problem is unbounded.
Hence the changed problem is infeasible. For this specific case, we can deduce that the
changed problem is infeasible directly since the first constraint is replaced by
x1 + x2 + x3 ≤ −3,
Suppose that a new activity xn+1 with cost cn+1 and consumption column an+1 is con-
sidered for possible production. Without resolving the problem, we can easily determine
whether producing xn+1 is worthwhile. First calculate zn+1 − cn+1 .
• If zn+1 − cn+1 > 0, then xn+1 is introduced into the basis and the simplex method
continues to find the new optimal solution.
161
9.3. ADDING A NEW ACTIVITY
Minimise −2x1 + x2 − x3
s.t. x1 + x2 + x3 ≤ 6,
−x1 + 2x2 ≤ 4,
x1 , x2 , x3 ≥ 0.
x1 x2 x3 x4 x5 RHS
f 0 −3 −1 −2 0 −12
x1 1 1 1 1 0 6
x5 0 3 1 1 1 10
Let’s find new optimal solution if a new activity x6 ≥ 0 with c6 = 1 and a6 = (−1, 2)T is
introduced.
From the optimal tableau, we find that the the inverse of the optimal basis B is
1 0
B −1 = .
1 1
We compute
−1 1 0
z6 − c6 = cB B −1 a6 − c6 = −2 0 − 1 = 1 > 0.
1 1 2
1 0 −1 −1
y6 = B −1 a6 = .
1 1 2 1
x1 x2 x3 x4 x5 x6 RHS
f 0 −3 −1 −2 0 1 −12
x1 1 1 1 1 0 -1 6
x5 0 3 1 1 1 1∗ 10
We continue using the simplex method: x6 enters the basis and x5 leaves. By pivoting
around the pivoting entry, which is marked with a star on the above tableau, we obtain
the following tableau
162
9.4. ADDING A NEW CONSTRAINT
x1 x2 x3 x4 x5 x6 RHS
f 0 −6 −2 −3 -1 0 −22
x1 1 4 2 2 1 0 16
x5 0 3 1 1 1 1 10
This is an optimal solution. Thus the optimal solution to the changed problem is (x∗1 , x∗2 , x∗3 ) =
(16, 0, 0), the optimal value is −22.
• If the optimal solution to the original problem satisfies the added constraint, it is
then obvious that the point is also an optimal solution to the new problem.
• If the optimal solution to the original problem does not satisfy the new constraint,
i.e., if the constraint cut away the optimal point, we can use the dual simplex method
to find the new optimal solution.
am+1 x ≤ bm+1
f + (cB B −1 N − cN )xN = cB B −1 b,
xB + B −1 N xN = B −1 b.
am+1
B xB + am+1
N xN + xn+1 = bm+1 .
163
9.4. ADDING A NEW CONSTRAINT
xB = B −1 b
xN = 0
The only possible violation of optimality of the new problem is the sign of bm+1 −am+1
B B −1 b.
If
bm+1 − am+1
B B −1 b ≥ 0,
bm+1 − am+1
B B −1 b < 0,
Minimise −2x1 + x2 − x3
s.t. x1 + x2 + x3 ≤ 6,
−x1 + 2x2 ≤ 4,
x1 , x2 , x3 ≥ 0.
x1 x2 x3 x4 x5 RHS
f 0 −3 −1 −2 0 −12
x1 1 1 1 1 0 6
x5 0 3 1 1 1 10
164
9.4. ADDING A NEW CONSTRAINT
−x1 + 2x3 ≥ 2.
Clearly the optimal point (x1 , x2 , x3 ) = (6, 0, 0) does not satisfy this constraint. The
constraint −x1 + 2x2 ≥ 2 is rewritten as
x1 − 2x3 + x6 = −2,
where x6 is a nonnegative slack variable. This row is added to the optimal tableau of the
above problem to obtain the following tableau.
x1 x2 x3 x4 x5 x6 RHS
f 0 −3 −1 −2 0 0 −12
x1 1 1 1 1 0 0 6
x5 0 3 1 1 1 0 10
x6 1 0 −2 0 0 1 −2
Multiply row 1 by −1 and add to row 3 in order to restore column x1 to a unit vector.
x1 x2 x3 x4 x5 x6 RHS
f 0 −3 −1 −2 0 0 −12
x1 1 1 1 1 0 0 6
x5 0 3 1 1 1 0 10
x6 0 −1 −3∗ −1 0 1 −8
The dual simplex method can then be applied to the above tableau. By minimum ratio test
of the dual simplex method, we see that x3 enters the basis to replace x6 . Thus, we get the
following tableau.
x1 x2 x3 x4 x5 x6 RHS
f 0 −8/3 0 −5/3 0 −1/3 −28/3
x1 1 2/3 0 2/3 0 1/3 10/3
x5 0 8/3 0 2/3 1 1/3 22/3
x3 0 1/3 1 1/3 0 −1/3 8/3
165
9.4. ADDING A NEW CONSTRAINT
In summary, when a constraint is added, restore the simplex tableau first, and then
continue simplex iterations (by using dual simplex method) until new solution is found.
166
Chapter 10
The simplex method assumes that an initial basic feasible solution (BFS) is at hand.
In many cases, such a BFS is not readily available. There are some work may be needed
to get the simplex method started.
1.
x1 + 2x2 ≤ 4,
−x1 + x2 ≤ 1,
x1 , x2 ≥ 0.
2.
x1 + x2 + x3 ≤ 6,
x1 , x2 , x3 ≥ 0.
For the first group of constraints, an initial basis can be found without any difficulty.
However, the second one is not. Some extra work is needed to get the initial basis.
In this chapter we will study two procedures that can be used to construct initial basis:
167
10.1. THE BIG-M METHOD
Both involve artificial variables to obtain an initial basic feasible solution. The ”Big M”
refers to a large number associated with the artificial variables, represented by the letter
M. As suggested by its names, the two-phase method consists of two phases. Phase I is
used to find a basic feasible solution for the original constraints, then the original linear
programming problem will be solved using the simplex method.
At the end of this chapter, students should be able to apply the big-M method and
the two-phase method to solve some LPs.
Observation:
Minimise cx
subject to Ax = b
x ≥ 0,
When we use the simplex method, we also assume that we have this form.
The big-M approach is to get rid of the artificial by assigning coefficients for those
artificial variables in the original objective function in such way as to make their presence
in the basis at a positive level very unattractive from the objective function point of view.
Consider the following problem:
Minimize cx + M (eT xa )
subject to Ax + xa = b,
x, xa ≥ 0.
168
10.1. THE BIG-M METHOD
Therefore, the simplex method will try to get the artificial variables out the basis,
and then continue to find an optimal solution of the original problem.
Two cases may arise after solving the Big-M problem by the simplex method:
Case A: The Big-M problem has an finite optimal solution. There are two subcases:
• (x, xa ) = (x∗ , 0) is an optimal solution of the Big-M problem. Then x∗ is the optimal
solution to the original problem.
Proof. In fact, if x is an arbitrary feasible point of the original LP, then (x, 0) is a
feasible solution to the Big-M problem. Since (x∗ , 0) is the optimal solution to the
Big-M problem, we have
cT x∗ + 0 ≤ cT x + 0,
i.e.,
cT x ∗ ≤ cT x
In this case, the original LP has no feasible point, i.e., the problem is infeasible.
(Why?)
• zk − ck = max(zj − cj ) > 0 and yk ≤ 0, and all the artificial are equal to zero, then
the original LP is unbounded. (Why?)
169
10.1. THE BIG-M METHOD
• zk − ck = max(zj − cj ) > 0 and yk ≤ 0, and not all the artificial are equal to zero,
then the original LP is infeasible. (Why?)
Minimise x1 − 2x2
subject to x1 + x2 ≥ 2,
x2 ≤ 3,
x1 , x2 ≥ 0.
Adding slack variables x3 and x4 and artificial variables x5 , the Big-M LP can be written
as
Minimise x1 − 2x2 + M x5
subject to x1 + x2 − x3 + x5 = 2,
x2 + x4 = 3,
x1 , x2 , x3 , x4 , x5 ≥ 0.
Note that in the simplex method, we work with tableau (including the initial one) of the
form
xB xN RHS
f 0 cB B −1 N − cN cB B −1 b
xB I B −1 N B −1 b
where the reduce cost zj − cj are zero for all basic variables.
There are two ways to find the initial tableau. The first approach is to compute all
170
10.1. THE BIG-M METHOD
Thus we obtain the following initial tableau for the simplex method
x1 x2 x3 x4 x5 RHS
f M − 1 M+2 −M 0 0 2M
x5 1 1∗ −1 0 1 2
x4 0 1 0 1 0 3
xB xN RHS
f −cB −cN 0
xB I B −1 N B −1 b
which encodes the formulation (8.1)-(8.2) in Section 8.2. Then we do elementary row
operations to reduce zj − cj for all basic variables to zero. In this example, we can start
with the following tableau
x1 x2 x3 x4 x5 RHS
f −1 2 0 0 −M 0
x5 1 1 −1 0 1 2
x4 0 1 0 1 0 3
Then we reduce zj − cj for all basic variables, x5 and x4 , to zero by a few elementary
row operations. Substituting the f -row by it plus the x5 -row multiplied by M , leads to the
following tableau:
x1 x2 x3 x4 x5 RHS
f M − 1 M+2 −M 0 0 2M
x5 1 1∗ −1 0 1 2
x4 0 1 0 1 0 3
171
10.1. THE BIG-M METHOD
This is exactly the initial tableau for the simplex method obtained in the first approach.
x2 enters and x5 leaves:
x1 x2 x3 x4 x5 RHS
f −3 0 2 0 −(M + 2) −4
x2 1 1 −1 0 1 2
x4 −1 0 1∗ 1 −1 1
x1 x2 x3 x4 x5 RHS
f −1 0 0 −2 −M −6
x2 0 1 0 1 0 3
x3 −1 0 1 1 −1 1
subject to x1 + x2 + 2x3 ≤ 4,
−x1 + x3 ≥ 4,
x3 ≥ 3,
x1 , x2 , x3 ≥ 0.
172
10.2. THE TWO-PHASE METHOD
Minimize −x1 − x2
subject to x1 − x2 − x3 = 1,
−x1 + x2 + 2x3 − x4 = 1,
x1 , x2 , x3 , x4 ≥ 0.
To find an initial basic feasible solution, one approach is Phase I problem which is
used to find a basic feasible solution for the original constraints, then we can initiate Phase
II wherein the Simplex Method is applied to solve the original linear programming prob-
lem. The combination of Phases I and II gives rise to the Two-Phase Simplex Method.
Phase 1. Solve the following linear programming problem starting with the basic
feasible solution x = 0 and xa = b :
Minimise eT xa
subject to Ax + xa = b,
x, xa ≥ 0.
Phase II. Solve the following linear programming problem starting with the basic
feasible solution xB = B −1 b and xN = 0 :
Minimise cB x B + cN x N
subject to xB + B −1 N xN = B −1 b,
xB , xN ≥ 0.
173
10.2. THE TWO-PHASE METHOD
Ax + xa = b, x ≥ 0, xa ≥ 0.
This gives an immediate basic feasible solution of the new system, namely xa = b and
x = 0. Even though we now have a starting basic feasible solution and the simplex method
can be applied, we have in fact changed the problem. In order to get back to our original
problem, we must force these artificial variables to zero, because Ax = b if and only if
Ax + xa = b with xa = 0. In other words, artificial variables are only a tool to get the
simplex method started, however, we must guarantee that these variables will eventually
drop to zero.
Minimise x1 − 2x2
subject to x1 + x2 ≥ 2,
−x1 + x2 ≥ 1,
x2 ≤ 3,
x1 , x2 ≥ 0.
Minimise x1 − 2x2
subject to x1 + x 2 − x3 = 2,
−x1 + x2 − x4 = 1,
x2 + x5 = 3,
x1 , x2 , x3 , x4 , x5 ≥ 0.
Phase I
174
10.2. THE TWO-PHASE METHOD
Minimise x6 + x7
subject to x1 + x2 − x3 + x6 =2
−x1 + x2 − x4 + x7 = 1
x2 + x5 =3
x1 , x2 , x3 , x4 , x5 , x6 , x7 ≥ 0.
x1 x2 x3 x4 x5 x6 x7 RHS
f0 0 0 0 0 0 −1 −1 0
x6 1 1 −1 0 0 1 0 2
x7 −1 1 0 −1 0 0 1 1
x5 0 1 0 0 1 0 0 3
First, restore the simplex tableau. Since x5 , x6 , x7 are basic variables, we may add rows 1,
2 to row 0 so that z6 − c6 = z7 − c7 = 0.
x1 x2 x3 x4 x5 x6 x7 RHS
f0 0 2 −1 −1 0 0 0 3
x6 1 1 −1 0 0 1 0 2
x7 −1 1* 0 −1 0 0 1 1
x5 0 1 0 0 1 0 0 3
x1 x2 x3 x4 x5 x6 x7 RHS
f0 2 0 −1 1 0 0 −2 1
x6 2* 0 −1 1 0 1 −1 1
x2 −1 1 0 −1 0 0 1 1
x5 1 0 0 1 1 0 −1 2
Replacing x6 by x1 , we have
175
10.2. THE TWO-PHASE METHOD
x1 x2 x3 x4 x5 x6 x7 RHS
f0 0 0 0 0 0 −1 −1 0
x1 1 0 −1/2 1/2 0 1/2 −1/2 1/2
x2 0 1 −1/2 −1/2 0 1/2 1/2 3/2
x5 0 0 1/2 1/2 1 −1/2 −1/2 3/2
which is the optimal tableau of the Phase I. There is no artificial variable in the optimal
basis of the first phase. We have a starting basic feasible solution for proceeding to the
second phase.
Phase II
Removing the artificial variables from the table, we do not consider it any more.
Replace the first row by the coefficients of the equation f − x1 + 2x2 = 0 since the original
objective is f = x1 − 2x2 .
x1 x2 x3 x4 x5 RHS
f −1 2 0 0 0 0
x1 1 0 −1/2 1/2 0 1/2
x2 0 1 −1/2 −1/2 0 3/2
x5 0 0 1/2 1/2 1 3/2
x1 x2 x3 x4 x5 RHS
f 0 0 1/2 3/2 0 −5/2
x1 1 0 −1/2 (1/2)* 0 1/2
x2 0 1 −1/2 −1/2 0 3/2
x5 0 0 1/2 1/2 1 3/2
Since z4 − c4 = 3/2 > 0, x4 is the entering variable. By minimum ratio test, x1 is the
leaving variable.
176
10.2. THE TWO-PHASE METHOD
x1 x2 x3 x4 x5 RHS
f −3 0 2 0 0 −4
x4 2 0 −1 1 0 1
x2 1 1 −1 0 0 2
x5 −1 0 1* 0 1 1
f x1 x2 x3 x4 x5 RHS
f 1 −1 0 0 0 −2 −6
x4 0 1 0 0 1 1 2
x2 0 0 1 0 0 1 3
x3 0 −1 0 1 0 1 1
Note that phase I moves from the infeasible point (0,0) to the infeasible point (0,1),
and finally to the feasible point (1/2, 3/2). From this extreme point, phase II moves to
the feasible point (0,2) and finally to the optimal point (0, 3).
• If all artificial variables are out of the basis, remove from the last form the artificial
and construct the initial simplex step since there is a basic feasible solution at hand,
and hence proceed to Phase II.
177
10.2. THE TWO-PHASE METHOD
• If some artificial variables are in the basis at zero level, we have two ways to continue.
We may first eliminate the columns corresponding to the nonbasic artificial variables
of Phase I, and then proceed to phase II directly to continue to replace those left
artificial variables without increasing the original objective values.
s.t. x1 + 2x2 + x3 ≥ 3,
2x1 − x2 + 3x3 ≥ 4,
x1 , x2 , x3 ≥ 0.
First, we introduce slack variables x4 and x5 , to write the problem in the standard form
s.t. x1 + 2x2 + x3 − x4 = 3,
2x1 − x2 + 3x3 − x5 = 4,
x1 , ..., x5 ≥ 0.
Phase I: now we introduce artificial variables x6 and x7 and consider the following phase-I
problem
Minimise x6 + x7
s.t. x1 + 2x2 + x3 − x4 + x6 = 3,
2x1 − x2 + 3x3 − x5 + x7 = 4,
x1 , ..., x7 ≥ 0.
x1 x2 x3 x4 x5 x6 x7 RHS
f0 0 0 0 0 0 −1 −1 0
x6 1 2 1 -1 0 1 0 3
x7 2 1 3 0 -1 0 1 4
x1 x2 x3 x4 x5 x6 x7 RHS
f0 3 1 4 -1 -1 0 0 7
x6 1 2 1 -1 0 1 0 3
x7 2 1 3∗ 0 -1 0 1 4
178
10.2. THE TWO-PHASE METHOD
x3 enters, x7 leaves
x1 x2 x3 x4 x5 x6 x7 RHS
f0 1/3 7/3 0 -1 1/3 0 -4/3 5/3
x6 1/3 7/3 0 -1 1/3 1 -1/3 5/3
x3 2/3 -1/3 1 0 -1/3 0 1/3 4/3
x2 enters, x6 leaves.
x1 x2 x3 x4 x5 x6 x7 RHS
f0 0 0 0 0 0 -1 -1 0
x2 1/7 1 0 -3/7 1/7 3/7 -1/7 5/7
x3 5/7 0 1 -1/7 -2/7 1/7 3/7 11/7
This is an optimal solution to the phase I problem. The artificial variables x6 and x7 are
not basic variable at optimality. Thus we remove columns x6 and x7 and move to phase
II.
Phase II: we replace the original cost function to obtain the tableau:
x1 x2 x3 x4 x5 RHS
f -2 -3 -4 0 0 0
x2 1/7 1 0 -3/7 1/7 5/7
x3 5/7 0 1 -1/7 -2/7 11/7
x1 x2 x3 x4 x5 RHS
f 9/7 0 0 -13/7 -5/7 59/7
x2 1/7 1 0 -3/7 1/7 5/7
x3 5/7∗ 0 1 -1/7 -2/7 11/7
x1 enters, x3 leaves.
x1 x2 x3 x4 x5 RHS
f 0 0 −9/5 −8/5 −1/5 28/5
x2 0 1 −1/5 −2/5 1/5 2/5
x1 1 0 7/5 −1/5 −2/5 11/5
179
10.2. THE TWO-PHASE METHOD
180
Chapter 11
In this chapter, we study another extension/variant of the simplex method, the dual
simplex method. This method is useful and necessary in, for instances, the following
situations.
• However, in these same instances, it is often possible to find a starting basis, which
is not necessary to be primal feasible, but which is dual feasible (i.e., all zj − cj ≤ 0
for minimization problem).
• The dual simplex method is a variant of the simplex method that would produce a
series of simplex tableau that maintain dual feasibility and complementary slackness
and strive toward primal feasibility.
• The dual simplex method solves the dual problem directly on the (primal) simplex
tableau. At each iteration we move from a basic feasible solution of the dual problem
to an improved basic feasible solution until optimality of the dual (and also the
primal) is reached, or else until we conclude that the dual is unbounded and that
the primal is infeasible.
181
11.1. DUAL INFORMATION FROM THE (PRIMAL) OPTIMAL SIMPLEX
TABLEAU
Minimise cx
subject to Ax ≥ b
x ≥ 0.
The slack variables are xn+1 , ..., xn+m . Let’s assume that we have the following optimal
simplex tableau:
F̌rom this optimal tableau, we may also get the optimal solution to the dual problem.
In fact, define
w = cB B −1 .
Then we have
zj − cj = cB B −1 aj − cj = waj − cj , j = 1, ..., n.
zj − cj ≤ 0, j = 1, ..., n
i.e., wA ≤ c.
Ňote that
an+i = −ei , cn+i = 0, i = 1, ..., m
182
11.1. DUAL INFORMATION FROM THE (PRIMAL) OPTIMAL SIMPLEX
TABLEAU
It follows from zn+i − cn+i ≤ 0 that w ≥ 0. Therefore, w is feasible to the dual problem,
and by denoting
w∗ = cB B −1 ,
bT w∗ = (cB B −1 )T b = cB b̄
i.e., the primal and dual objectives are equal. By weak duality, we conclude that w∗ is
dual optimal.
Theorem 11.1. At optimality of the primal minimization problem in the canonical form,
w∗ = cB B −1
for i = 1, ..., m.
W̌hen the optimal solution is attained, from the final tableau of the simplex method,
we may get both the primal and dual optimal solutions. by using the following correspon-
dence
2. The variables of the primal problem correspond to the dual slack variables.
Note that a symmetric analysis holds for the equality constrained case.
w∗ = cB B −1 ,
183
11.2. THE DUAL SIMPLEX METHOD
min{cx : Ax = b, x ≥ 0},
and suppose that we have a dual feasible point, i.e., the initial table of simplex method
is given as follows,
Where
zj − cj ≤ 0, i = 1, ...., n.
Case 1. If b̄ ≥ 0, then the current point is already optimal to the linear program.
Case 2. If b̄r < 0 for some r, and the rth row vector is nonnegative, i.e., yrj ≥ 0,
for all j = 1, ..., n, then the dual is unbounded (and hence the primal problem is infeasible).
Case 3. If b̄r < 0 for some r, and there exists at least one element yrj < 0 in the rth
row. In this case, let xBr leave, and use the following minimum ratio test to determine
the variable to enter:
zk − ck zj − cj
= min : yrj < 0 .
yrk yrj
Ǎfter pivoting at yrk , it is easy to see that the new
yrj
(zj − cj )0 = (zj − cj ) − (zk − ck ) ≤ 0,
yrk
184
11.2. THE DUAL SIMPLEX METHOD
and hence the dual feasibility is kept. Furthermore, we may show that the dual objective
is improved, in fact, the new objective value is
Ťhus, after a pivoting, the dual simplex method moves from one dual basic feasible
point to another one with improved function value. Under non-degeneracy assumption,
the algorithm terminates after finite iterations and finds the primal and dual optimal
solutions.
Initialization Step: Find a basis B of the primal such that zj − cj ≤ 0 for all j.
Main Step:
Step 2. If yrj ≥ 0 for all j, stop, the dual is unbounded (and hence the primal is
infeasible). Otherwise, select the pivot column k by the following minimum ratio test:
zk − ck zj − cj
= min : yrj < 0 .
yrk yrj
185
11.2. THE DUAL SIMPLEX METHOD
s.t. x1 + 2x2 + x3 ≥ 3,
2x1 − x2 + 3x3 ≥ 4,
x1 , x2 , x3 ≥ 0.
Multiplying both sides of the inequalities by −1 and adding slack variables x4 and x5 , we
may form the first table as follows:
x1 x2 x3 x4 x5 RHS
f −2 −3 −4 0 0 0
x4 −1 −2 −1 1 0 −3
x5 −2∗ 1 −3 0 1 −4
Since b2 = −4 < 0, x5 leaves the basis. Which one to enter the basis? By minimum
ratio test,
z1 − c1 z3 − c3 −2 −4
min{ , } = min{ , } = min{1, 4/3} = 1
y21 y23 −2 −3
which indicates that x1 enters the basis. So after pivoting operations, we get the next
tableau.
x1 x2 x3 x4 x5 RHS
f 0 −4 −1 0 −1 4
x4 0 −(5/2)∗ 1/2 1 −1/2 −1
x1 1 −1/2 3/2 0 −1/2 2
Since b̄1 = −1 < 0, x4 must leave the basis. By the minimum ratio test, x2 enters the
basis. After pivoting, we get the optimal tableau
x1 x2 x3 x4 x5 RHS
f 0 0 −9/5 −8/5 −1/5 28/5
x2 0 1 −1/5 −2/5 1/5 2/5
x1 1 0 7/5 −1/5 −2/5 11/5
Thus the optimal solution is
x∗ = (11/5, 2/5, 0),
186
Chapter 12
Practice exercises
(b)
f : R3 → R, for x = (x1 , x2 , x3 ) ∈ R3 .
1
f (x) = (x51 +x52 −3x21 x2 x23 +x21 x22 x3 +3x41 x2 +x53 ) 5
(c)
f : R3×3 → R, for A ∈ R3×3 .
1
f (A) = (det(A)) 3 ,
(c)
f : R3×3 → R, f (A) = tr(A) + (det(A)) 3 , for A ∈ R3×3 .
1
(e)
f : Pn (R) → R, for p = a0 + a1 x + . . . an xn ,
1
f (p) = (a0 a1 · · · a5 ) 6
For each of the above function, answer the following questions (below U is the corre-
sponding vector space in each case):
187
12.1. EXERCISES FOR CHAPTER 1
3. Is f a linear map?
Q 2. Consider the following functions (note that in each case, C is a vector space over
itself)
(a)
f : C → C, f (z) = z̄ for z ∈ C.
(b)
f : C → C, f (z) = Rel(z) for z ∈ C.
2. Is f a linear map?
2. What is the matrix representation of f with respect to the standard basis of R2×2 .
Q 5. Let A, B, C be three different bases of the same finite-dimensional vector space. Let
P be the transition matrix from A to B, and Q is the transition matrix from B to C.
What is the transition matrix from A to C? Prove your finding.
188
12.1. EXERCISES FOR CHAPTER 1
189
12.2. EXERCISES FOR CHAPTER 2
0.4 0.2
Q 11. Consider a 2 × 2 matrix A = .
0.6 0.8
Q 12. Let V be an inner product space. Show that for all u, v ∈ U the following identities
hold true
ku + vk2 + ku − vk2 = 2(kuk2 + kvk2 ),
and
1h 2 2
i
hu, vi = ku + vk − ku − vk .
4
Q 13. Suppose that {u1 , . . . , un } is a set of orthonormal vectors in an inner product space
V . Find ku1 + . . . + un k.
Q 14. Using the Cauchy-Schwarz inequality show that if a1 , . . . , an are positive real num-
bers then
1 1
(a1 + . . . + an ) + ... + ≥ n2 .
a1 an
Q 15. Let V be an inner product space. Let u, u1 , . . . , un ∈ V . Show that if u is orthogonal
to u1 , . . . , un then it is orthogonal to the subspace spanned by u1 , . . . , un .
Apply the Gram-Schmidt process to transform the standard basis {1, x, x2 } to an orthono-
mal basis.
190
12.3. EXERCISES FOR CHAPTERS 3 & 4
Q 17. Is the following problem a linear programming problem? If not, formulate it into
a linear programming problem.
x1 + x2 ≤ 1,
x1 , x2 ≥ 0.
Q 18. Is the following problem a linear programming problem? If not, formulate it into
a linear programming problem.
c1 x + c2 y + c3
Minimize
c4 x + c5 y + d
subject to a1 x + a2 y ≤ b1 ,
a3 x + a4 y ≥ b 2 ,
c3 x + c4 y + d > 0.
Maximize 2x1 + x2
x1 + x2 ≤ 5,
3x1 − 2x2 ≤ 6,
x1 , x2 ≥ 0.
Q 20. Show that the following LP problem has infinitely many optimal solutions
Minimize 2x1 + x2
subject to x2 ≥ 3,
x1 ≤ 2,
x1 , x2 ≥ 0.
191
12.4. EXERCISES FOR CHAPTER 5
Minimize 2x1 + x2
subject to x2 ≥ 3,
x1 ≤ 2,
x1 + 2x2 ≤ 4,
x1 , x2 ≥ 0.
Q 23. Show that the Cartesian product of two convex sets is also a convex set.
Q 24. Let f : Rn → R be a convex function. Show that the sublevel sets {x : f (x) ≤ a}
and {x : f (x) < a} with a ∈ R are convex sets.
3. Compute the values of the function 2x + 5y at the extreme points of A, and compare
the maximum one with the result in part 1.
A = {(x, y) ∈ R2 : −x + y ≤ 4, 2x + y ≤ 5, x + 2y ≥ 2, 3x − 2y ≤ 6, x ≥ 0, y ≥ 0}.
192
12.5. EXERCISES FOR CHAPTER 6
3. Compute the values of the function x − 5y at the extreme points of A, and compare
the minimum one with the result in part 1.
Q 28. Find the dual problems for the following linear programming problems
x1 + x2 − 3x3 ≤ 5,
x1 , x2 , x3 ≥ 0.
x1 + x2 − 2x3 ≤ 5,
x1 , x2 , x3 ≥ 0.
x1 + x2 ≤ 5,
x1 − 2x2 ≤ 4,
x1 , x2 ≥ 0.
3. Using duality theorems and complementary slackness conditions to solve the dual
problem.
193
12.6. EXERCISES FOR CHAPTERS 7 & 8 & 9
x1 + 2x2 + x3 ≤ 4,
x1 + x2 + x3 ≤ 5,
x1 , x2 , x3 ≥ 0.
3. In each of the following change of the data, determine if the optimal solution is
affected. If it is affected, find the new optimal solution
(a) The cost coefficient vector is changed to (c01 , c02 , c03 ) = (−2, 4, 1).
(b) The right hand side vector is changed to (b01 , b02 , b03 ) = (6, 4, 6).
(c) The right hand side vector is changed to (b01 , b02 , b03 ) = (6, 6, 5).
Maximize x + 2y
subject to 2x + 3y ≤ 6,
2x + y ≤ 4,
x ≤ 5,
x, y ≥ 0.
To understand how the optimal solution to the above problem is affected when the data is
changed, solve the following problems.
Maximize x + (2 + a)y
subject to 2x + 3y ≤ 6,
2x + y ≤ 4,
x ≤ 5,
x, y ≥ 0.
194
12.7. EXERCISES FOR CHAPTERS 10 & 11
Maximize x + 2y
subject to 2x + 3y ≤ 6 + a,
2x + y ≤ 4,
x ≤ 5,
x, y ≥ 0.
You are encouraged to create your own problems by changing other data. Note that this
problem has only two variables, you can solve it and observe the change geometrically.
Q 32. Solve the following LP problems using any of the method you have studied.
s.t. x1 + 2x2 + x3 ≥ 3,
x1 , x2 , x3 ≥ 0.
s.t. x1 + 2x2 + x3 ≥ 3,
x1 , x2 , x3 ≥ 0.
You are encouraged to create your own similar problems where the constraints are mixed.
195
Index
196
INDEX
197
Bibliography
[Bea55] E. M. L. Beale. Cycling in the dual simplex algorithm. Naval Research Logistics
Quarterly, pages 269–275, 1955.
[Gre94] Harvey J. Greenberg. The use of the optimal partition in a linear programming
solution for postoptimal analysis. Operations Research Letters, 15(4):179 – 185,
1994.
[KM72] V. Klee and G.J. Minty. How good is the simplex algorithm? In O. Shisha,
editor, Inequalities, III, pages 159–175, 1972.
[KW98] R.W. Kaye and R. Wilson. Linear Algebra. Oxford science publications. Oxford
University Press, 1998.
[Sma81] Steve Smale. The fundamental theorem of algebra and complexity theory. Bull.
Amer. Math. Soc. (N.S.), 4(1):1–36, 01 1981.
[Str06] Gilbert Strang. Linear algebra and its applications. Thomson, Brooks/Cole,
2006.
[Tao16] T. Tao. Linear algebra course notes. AMS open math notes, 2016.
198