Week 8

1 LP is easy: the Ellipsoid Method

In 1979 Khachyan proved that LP is solvable in polynomial time by a method of
shrinking ellipsoids. The running time is polynomial in the number of variables
n, the number of constraints m and the logarithm of the largest coefficient log U .

Khachyan developed his method for finding a feasible point in P = {x IRn |

Ax b} or decide that P = . Notice that solving an LP optimization problem
is a matter of finding a feasible solution to the system of primal inequalities,
dual inequalities and the equality that primal and dual objective values are
equal (see Section 8.4 in [B&T]).

The idea behind the ellipsoid method is rather simply explained in a two-
dimensional picture (on the blackboard). The proof that it works is technical
and based on several ingredients. In what follows let A and b be integer and let
U be the maximum absolute value of the entries in A and b..
(a) We can guess a ball that contains all extreme points of P , and the volume
of which is not too large in terms of n, m and log U ;
(b) If P is non-empty then its volume is not too small in terms of n, m and
log U ;
(c) In each step either the center of the ellipsoid is in P or the next ellipsoid
contains P , and in fact the entire half of the previous ellipsoid containing
P;
(d) In each step the ratio between the volume of the next ellipsoid and the
volume of the previous ellipsoid is large enough in terms of n, m and log U .

1.1 An iteration
We shall first be concerned in showing (c) and (d). We will obtain perfect in-
sight by considering how this works for an ellipsoid that contains the half of the
unit ball E0 consisting of all points with non-negative first coordinate; that is
the unit ball E0 intersected with the halfspace H0 = {x IRn | eT1 x 0}. Any
general case can in fact be transformed into this result.

Lets start with a general description of an ellipsoid. For any positive definite
symmetric matrix D and centre z an ellipsoid is defined by
E(z, D) = x IRn | (x z)T D1 (x z) 1 .


Remember a matrix is positive definite if xT Dx > 0 x 6= 0. D positive defi-

nite D1 positive definite. Also D is non-singular. Any such matrix can be

1
written as the product of two symmetric non-singular matrices D = D1/2 D1/2 .

So, within this notation the unit ball is

E0 = E(0, I),
and we will verify that the following ellipsoid contains E0 H0 .
E00 = E(z, D) = x IRn | (x z)T D1 (x z) 1 ,


using e1 to denote the first unit vector in IRn ,

e1 1
z = (= ( , 0, . . . , 0))
n+1 n+1
and
n2
 
2
D = 2 I e1 eT1 ,
n 1 n+1
or written out
2

1 n+1 0 . . . . 0

0 1 0 . . . 0

n 2

. 0 1 0 . . .

D = 2 . . . . . . . .
n 1

. . . . . . .

. . . . . 1 0
0 0 . . . 0 1
Let us check that this indeed contains all the extreme points of E0 H0 : i.e.,
all points on the n 1-dimensional unit ball of points with x1 = 0, and the
point (1, 0, 0, . . . , 0). A little thought should make it clear that such an ellipsoid
contains the entire half-ball E0 H0 . It is in fact the smallest ellipsoid with this
property, which I wont show, because for the rest of the story it is irrelevant.
Written out, the ellipsoid becomes
( 2  2 n
)
n2 1 X 2

0 n n+1 1
E0 = x IR | x1 + x 1 . (1)
n n+1 n2 i=2 i
Pn
The points in E00 with x1 = 0 and i=2 x2i = 1 satisfy
2  2
n2 1 n2 1

n+1 1 1
+ = + = 1,
n n+1 n2 n2 n2
The point (1, 0, . . . , 0) satisfies
 2  2
n+1 1
1 = 1.
n n+1

So, we would like to know the ratio V olume(E00 )/V olume(E0 ). We do this by
making an appropriate transformation F such that F (E00 ) = E0 , which allows
us to use the following lemma.

2
Lemma 1.1 (Lemma 8.1 in [B&T]) Given the affine transformation S(x) =
Bx+b, with B a non-singular matrix, then V ol(S(L)) = | det(B)|V ol(L), where
S(L) = {y IRn | x L s.t. y = Bx + b}.
The following transformation works
F (x) = D1/2 (x z)
because
x E00 (x z)T D1 (x z) 1

(x z)T D1/2 D1/2 (x z) 1

D1/2 (x z) E0

F (x) E0 .
Applying Lemma 1.1
V olume(E0 ) = | det(D1/2 )| V olume(E00 ).
Using the property that, for any non-singular square matrix B,
1
| det(B 1/2 )| = p ,
| det(B)|
we get
q
V olume(E00 ) = det(D) V olume(E0 ).
p
det(D) is easily seen to be
n/2  1/2
n2

2
1 .
n2 1 n+1
Hence,
n/2  1/2
V olume(E00 )

n2 2
V olume(E0 ) = n2 1 1 n+1

 (n1)/2
n n2
= n+1 n2 1

  (n1)/2
1 1
= 1 n+1 1+ n2 1

applying 1 + a < ea , a 6= 0,
 2
(n1)/2
< e1/(n+1) e1/(n 1)

= e1/(2(n+1)) .

3
In general, let us be given ellipsoid

E = E(z, D) = x IRn | (x z)T D1 (x z) 1 .



and some halfspace

H = {x Rn | aT x aT z}

which is bounded by the hyperplane H = {x IRn | aT x = aT z} through the

centre z of E, hence containing exactly half of E. Then the following ellipsoid
contains E H.

E 0 = E(z, D)

with
1 Da
z = z +
n + 1 aT Da

and
n2 2 DaaT D
 
D = 2
D .
n 1 n + 1 aT Da

And by transforming this situation to the unit ball we obtain:

V olume(E 0 ) V olume(E00 )
= < e1/(2(n+1)) .
V olume(E) V olume(E0 )

1.2 The ellipsoid method for feasibility

(a) We can guess a ball that contains all extreme points of P , and the volume
of which is not too large in terms of n, m and log U ;
(b) In each step check if the center z of the ellipsoid E is in P . If so, we have a
feasible point. If not determine a constraint aT x b of P that is violated
by z. Take the halfspace H = {x Rn | aT x aT z} and determine the
next ellipsoid that E H, containing P .

(c) Check if the volume dropped below the critical value. If so, conclude that
there does not exist a feasible point: P = . Otherwise reiterate (b).

So we have now a formula to do (b) and we have a bound on the reduction in

volume in each iteration. Suppose that V is the volume of the initial ball and
v a lowerbound on the volume of a feasible P , then there exists a t such that
after at most t iterations we have found a point in P or the volume of the
ellipsoid has reduced to

V et /(2(n+1))
<v

4
and we correctly conclude that P is empty. Correct calculation gives that

t = d2(n + 1) log(V /v)e. (2)

Extremely coarse bounds on V and v follow rather straightforwardly from

bounds on each of the coordinates of any bfs, which in its turn follows through
Cramers rule. In case P is full-dimensional,
2
V (2n)n (nU )n .

and
2
v nn (nU )n (n+1)
.

There is indeed a little complication in case P is not full-dimensional, in which

case V olume(P ) = 0. It requires then some effort to show that we may perturb
P a little bit, in such a way that the perturbation is full-dimensional if P is
non-empty and such that it is empty if P is empty.

Filling in V and v in the expression (2) yields a number of iterations of O(n4 (log nU )).
In case P is not full-dimensional then the perturbed polytope would have dif-
ferent bounds on V and v resulting in O(n6 (log nU )) iterations.

It requires technicalities, which the book does not give, to show that the preci-
sion of taking square roots and of multiplications of numbers can be cut down
sufficiently to conclude that also the running time per iteration can be polyno-
mially bounded.

I leave studying how to adapt the method to solve optimisation i.o. feasibility
problems in Section 8.4 to yourself.

It is extremely important for what follows to notice that the number of iterations
is independent of m the number of constraints! It is only dependent on n, the
number of variables, and log U , the size of the largest coefficient.

1.3 The ellipsoid method and the separation problem

In each iteration of the ellipsoid method a restrictive form of the separation
problem is solved.
Separation Problem: Given polyhedron P IRn and x IRn :
(a) Either decide that x P , or
(b) Find a hyperplane separating x from P .
In the ellipsoid method the separating hyperplane is found among those parallel
to the constraints of P . Obviously, this problem can be solved in O(mn) time,
making the ellipsoid method run in time polynomial in n, m and log U . But the
number of iterations is independent of m (the number of restrictions). Thus,

5
if for a family of polyhedra we can solve the separation problem in time poly-
nomial in n and log U only, then we can solve the linear optimization problems
over this family in time polynomial in n and log U .

This is particularly interesting in case the number of restrictions is exponential

in the number of variables, and the restrictions do not have to be written out
explicitly to define the problem. Take for example the min-cut problem find-
ing a minimum s, t-cut in an undirected graph G = (N , E). The s, t-cuts are
characterized by
P
eK xe 1, K K

0 xe 1, e E,

where, K is the set of all s, t-paths in G. In fact you could see the formulation
as the LP-relaxation of a min-cut, but it can be shown that all extreme points
of this polytope are {0, 1}-vectors. The restrictions say that there should be
at least 1 edge on each s, t-path, clearly the definition of a path. We do not
wish to specify all these paths since, indeed, there are exponentially many in
the number of edges.

Given a {0, 1}-vector x, it is however a matter of deleting from the graph all
edges e with xe = 1 and use a simple graph search algorithm to decide if there
exists an s, t-path in the remaining graph.P If there is not, then x represents a
cut, otherwise one finds a path K with eK xe < 1 and therefore a violated
constraint. In fact this was Exercise 8.8.

Thus, since separation over the cut-polytope can be done in time polynomial
in the number of edges only, we can conclude immediately that the min-cut
problem can be solved in polynomial time using the ellipsoid method (though
we know it can be done faster).

Complexity equivalence of separation and optimization is extremely useful for

the construction of polynomial time approximation algorithms based on round-
ing optimal solutions of LP-relaxations of Integer LP problems. The bounds on
the integrality gap is usually smaller in ILP-formulations with an exponential
number of constraints. Let us briefly go through the example given in the book
for the famous travelling salesman problem (TSP).

In TSP we are given a graph G = (V, E) with edge costs ce , e E. We are to

determine a circuit in the graph of total minimum cost containing all vertices
of V . In the integer LP problem decision variables are xe , with xe = 1 if e is
selected in the tour and 0 otherwise. The following LP has as feasible region
the subtour elimination polytope, an LP-relaxation of the TSP. We use (S) to
denote the edges with exactly one if its two nodes in S. Then, (i) is the edges

6
incident to node i.
P
min eE ce xe
P
s.t. e(i) xe = 2, i
P
e(S) xe 2, S V
0 xe 1, e E.

The second set of constraints are the so-called subtour-elimination constraints.

Without these even the ILP-problem can be solved in polynomial time, giving
so-called 2-factors. These are a bunch of vertex disjoint cycles. If it is only 1
then that is the optimal TSP tour. Usually it is not and subtour elimination
constraints need to be added.

Clearly there are exponential many of them, so let us see how we can solve the
separation problem over the subtour elimination constraints.
P
If we interpret xe as the capacity of the edge e then e(S) xe can be inter-
preted as the size
Pof the cut S. Thus, if the subtour elimination constraint w.r.t.
S is violated if e(S) xe < 2, i.e. S is a cut of size less than 2. Hence, check-
ing if some subtour elimination constraint is violated is a matter of finding the
mincut in G with capacities on the edges xe , which can be done in polynomial
time.

The optimal solution of this LP-relaxation is usually not integer, but gives good
lower bounds. The value optimal of the LP-relaxation is conjectured to be
within 3/4th of the integer optimal value.

Material of Week 8 from [B& T]

Chapter 8

Exercises of Week 8
8.9