Discrete (And Continuous) Optimization WI4 131: November - December, A.D. 2004
Discrete (And Continuous) Optimization WI4 131: November - December, A.D. 2004
Discrete (And Continuous) Optimization WI4 131: November - December, A.D. 2004
WI4 131
Kees Roos
Technische Universiteit Delft
Faculteit Electrotechniek, Wiskunde en Informatica
Afdeling Informatie, Systemen en Algoritmiek
e-mail: [email protected]
URL: http://www.isa.ewi.tudelft.nl/roos
November December, A.D. 2004
Course Schedule
1. Formulations (18 pages)
2. Optimality, Relaxation, and Bounds (10 pages)
3. Well-solved Problems (13 pages)
4. Matching and Assigments (10 pages)
5. Dynamic Programming (11 pages)
6. Complexity and Problem Reduction (8 pages)
7. Branch and Bound (17 pages)
8. Cutting Plane Algorithms (21 pages)
9. Strong Valid Inequalities (22 pages)
10. Lagrangian Duality (14 pages)
11. Column Generation Algorithms (16 pages)
12. Heuristic Algorithms (15 pages)
13. From Theory to Solutions (20 pages)
Optimization Group 1
Capter 2
Optimality, Relaxation, and Bounds
Optimization Group 2
Optimality and Relaxation
(IP) z = max {c(x) : x X Z
n
}
Basic idea underlying methods for solving (IP): nd a lower bound z z and an upper
bound z z such that z = z = z.
Practically, this means that any algorithm will nd a decreasing sequence
z
1
> z
2
> . . . > z
s
z
of upper bounds, and an increasing sequence
z
1
< z
2
< . . . < z
t
z
of lower bounds. The stopping criterion in general takes the form
z
s
z
t
,
where epsilon is some suitably chosen small nonnegative number.
Optimization Group 3
How to obtain Bounds?
Every feasible solution x X provides a lower bound z = c(x) z.
This is essentially the only way to obtain lower bounds. For some IPs, it is easy to nd
a feasible solution (e.g. Assignment, TSP, Knapsack), but for other IPs, nding a feasible
solution may be very dicult.
The most important approach for nding upper bounds is by relaxation. The given IP is
replaced by a simpler problem whose optimal value is at least as large as z. There are two
obvious ways to get a relaxation:
(i) Enlarge the feasible set.
(ii) Replace the objective function by a function that has the same or a larger value every-
where.
Denition 1 The problem (RP) z
R
= max {f(x) : x T Z
n
} is a relaxation of
(IP) if X T and f(x) c(x) for all x X.
Proposition 1 If (RP) is a relaxation of (IP) then z
R
z.
Proof: If x
X T and z = c(x
) f(x
). As
x
T, f(x
) z
R
.
Optimization Group 4
Linear Relaxations
Denition 2 For the IP max
_
c
T
x : x X = P Z
n
_
with formulation P, a linear
relaxation is the LO problem z
LP
= max
_
c
T
x : x P
_
.
Recall that P =
_
x R
n
+
: Ax b
_
. As P Z
n
P and the objective is unchanged,
this is clearly a relaxation. Not surprisingly, better formulations give tighter (upper) bounds.
Proposition 2 If P
1
and P
2
are two formulations for the feasible set X in an IP, and
P
1
P
2
, the the respective upper bounds z
LP
i
(i = 1, 2) satisfy z
LP
1
z
LP
2
.
Sometimes the relaxation RP immediately solves the IP.
Proposition 3 If the relaxation (RP) is infeasible, then so is (IP). On the other hand, if
(RP) has an optimal solution x
and x
X and c(x
) = f(x
), then x
is an optimal
solution of (IP).
Proof: If (RP) is infeasible then T = . Since X T, also X = . For the second part
of the lemma: as x
X, z c(x
) = f(x
) = z
R
and z z
R
we get c(x
) = z.
Optimization Group 5
Example
z = max {4x y : 7x 2y 14, y 3, 2x 2y 3, x, y 0, x, y integer}
The gure below illustrates the situation.
0 1 2 3 4 5 6 7
0
1
2
3
4
5
6
7
c
x
y
The gure makes clear that x = 2, y = 1 is the optimal solution, with 7 as objective value.
The optimal solution of the LP relaxation is x =
20
7
, y = 3 with
59
7
as objective value. This
is an upper bound. Rounding this bound to an integer gives 8 as the linear relaxation bound.
Optimization Group 6
Combinatorial Relaxations
Whenever the relaxation is a combinatorial optimization problem, we speak of a combinatorial
relaxation. Below follow some examples.
TSP: The TSP, on a digraph D = (V, A), amounts to nding a (salesman, or Hamiltonian)
tour T with minimal length in terms of given arc weights c
ij
, (i, j) A. We have seen that
a tour is an assignment without subtours. So we have
z
TSP
= min
_
(i,j)T
c
ij
: T is a tour
_
_
min
_
(i,j)T
c
ij
: T is an assigment
_
_
.
Symmetric TSP: The STSP, on a graph G = (V, E), amounts to nding a tour T with
minimal length in terms of given edge weights c
e
, e E.
Denition 3 A 1-tree is a subgraph consisting of two edges adjacent to node 1, plus the
edges of a tree on the remaining nodes {2, . . . , n}.
Observe that a tour consists of two edges adjacent to node 1, plus the edges of a path through
the remaining nodes. Since a path is a special case of a tree we have
z
STSP
= min
_
_
_
eT
c
e
: T is a tour
_
_
_
min
_
_
_
eT
c
e
: T is a 1-tree
_
_
_
.
Optimization Group 7
Combinatorial Relaxations (cont.)
Quadratic 0-1 Problem: This is the (in general hard!) problem of maximizing a quadratic
function over the unit cube:
z = max
_
_
_
1i<jn
q
ij
x
i
x
j
+
1in
p
i
x
i
: x {0, 1}
n
_
_
_
Replacing all terms q
ij
x
i
x
j
with q
ij
< 0 by 0 the objective function does not decrease. So
we have the relaxation
z
R
= max
_
_
_
1i<jn
max
_
q
ij
, 0
_
x
i
x
j
+
1in
p
i
x
i
: x {0, 1}
n
_
_
_
This problem is also a quadratic 0-1 problem, but now the quadratic terms have nonnegative
coecients. Such a problem can be solved by solved by solving a series of (easy!) maximum
ow problems (see Chapter 9).
Knapsack Problem: The set underlying this problem is X =
_
_
_
x Z
N
+
:
n
j=1
a
j
x
j
b
_
_
_
.
This set can be extended to X
R
=
_
_
_
x Z
N
+
:
n
j=1
_
a
j
_
x
j
b
_
_
_
, where a denotes
the largest integer less than or equal to a.
Optimization Group 8
Lagrangian Relaxations
Consider the IP
(IP) z = max
_
c
T
x : Ax b, x X Z
n
_
.
If this problem is too dicult to solve directly, one possible way to proceed is to drop the
constraint Ax b. This enlarges the feasible region and so yields a relaxation of (IP).
An important extension of this idea is not just to drop complicating constraints, but then to
add them into the objective function with Lagrange multipliers.
Proposition 4 Suppose that (IP) has an optimal solution, and let u R
m
. Dene
z(u) = max
_
c
T
x +u
T
(b Ax) : x X
_
.
Then z(u) z for all u 0.
Proof: Let x
= z, Ax
b and x
X. Since
u 0, it follows that c
T
x
+u
T
(b Ax
) c
T
x
= z.
The main challenge is, of course, to nd Lagrange multipliers that minimize z(u). The best
Lagrange multipliers are found by solving (if we can!)
min
u0
z(u) = min
u0
max
xX
_
c
T
x +u
T
(b Ax)
_
.
Optimization Group 9
LO Duality (recapitulation)
Consider the LO problem (P) z = max
_
c
T
x : Ax b, x 0
_
.
Its dual problem is (D) w = min
_
b
T
y : A
T
y c, y 0
_
.
Proposition 5 (Weak duality) If x is feasible for (P) and y is feasible for (D) then
c
T
x b
T
y.
Proof: c
T
x
_
A
T
y
_
T
x = y
T
Ax y
T
b = b
T
y.
Proposition 6 (Strong duality) Let x be feasible for (P) and y feasible for (D). Then
x and y are optimal if and only if c
T
x = b
T
y.
Scheme for dualizing:
Primal problem (P) Dual problem (D)
min c
T
x max b
T
y
equality constraint free variable
inequality constraint variable 0
inequality constraint variable 0
free variable equality constraint
variable 0 inequality constraint
variable 0 inequality constraint
Optimization Group 10
Duality
In the case of Linear Optimization every feasible solution of the dual problem gives a bound
for the optimal value of the primal problem. It is natural to ask whether it is possible to nd
a dual problem for an IP.
Denition 4 The two problems
(IP) z = max {c(x) : x X} ; (D) w = min{(u) : u U}
form a (weak)-dual pair if c(x) (u) for all x X and all u U. When moreover
w = z, they form a strong-dual pair.
N.B. Any feasible solution of a dual problem yields an upper bound for IP. On the contrary,
for a relaxation only its optimal solution yields an upper bound.
Proposition 7 The IP z = max
_
c
T
x : Ax b, x Z
n
+
_
and the IP
w
LP
= min
_
u
T
b : A
T
u c, u Z
m
+
_
form a dual pair.
Proposition 8 Suppose that (IP) and (D) are a dual pair. If (D) is unbounded then
(IP) is infeasible. On the other hand, if x
X and u
U satisfy c(x
) = w(u
), then
x
_
_
_
_
_
_
_
Greedy Solution: Order the edges according to nondecreasing cost, and seek to use them
in this order. So we proceed as follows to construct the tour at the left.
3
4
2
5
1
6
Heuristic tour
step arc length
1 (1,3) 2 accept
2 (4,6) 3 accept
3 (3,6) 6 accept
4 (2,3) 7 conict in node 3
5 (1,4) 8 creates subtour
6 (1,2) 9 accept
7 (2,5) 10 accept
8 (4,5) 24 forced to accept
3
4
2
5
1
6
Better tour
The length of the created tour is z
G
= 54. The tour 1 4 6 5 2 3 1 is
shorter: it has length 49.
Optimization Group 15
Primal Bounds: Local Search
Local search methods assume that a feasible solution is known. It is called the incumbent.
The idea of a local search heuristic is to dene a neighborhood of solutions close to the
incumbent. Then the best solution in this neighborhood is found. If it is better than the
incumbent, it replaces it, and the procedure is repeated. Otherwise, the incumbent is locally
optimal with respect to the neighborhood, and the heuristic terminates.
Below we give two examples.
Optimization Group 16
Primal Bounds: Local Search (cont.)
Uncapacitated Facility Location: Consider an instance with m = 6 clients and n = 4 depots, and costs
as shown below:
(c
ij
) =
_
_
_
_
_
_
_
6 2 3 4
1 9 4 11
15 2 6 3
9 11 4 8
7 23 2 9
4 3 1 5
_
_
_
_
_
_
_
and (f
j
) = (21, 16, 11, 24)
Let N = {1, 2, 3, 4} denote the set of depots, and S the set of open depots. Let the incumbent be the
solution with depots 1 and 2 open, so S = {1, 2}. Each client is served by the open depot with cheapest cost
for the client. So the costs for the incumbent are (2 +1 +2 +9 +7 +3) +(21 +16) = 61.
A possible neighborhood Q(S) of S is the set of all solutions obtained from S by adding or removing a single
depot:
Q(S) = {T N : T = S {j} for j / S or T = S \ {i} for i S} .
In the current example: Q(S) = {{1} , {2} , {1, 2, 3} , {1, 2, 4}}. A simple computation makes clear that
the costs for these 4 solutions are 63, 66, 60 and 84, respectively. So S = {1, 2, 3} is the next incumbent.
The new neighborhood becomes Q(S) = {{1, 2} , {1, 3} , {2, 3} , {1, 2, 3, 4}}, with minimal costs 42 for
S = {2, 3}, which is the new incumbent.
The new neighborhood becomes Q(S) = {{2} , {3} , {1, 2, 3} , {2, 3, 4}}, with minimal costs 31 for
S = {3}, the new incumbent.
The new neighborhood becomes Q(S) = {{1, 3} , {2, 3} , {3, 4} , }, with all costs > 31. So S = {3} is
a locally optimal solution.
Optimization Group 17
Primal Bounds: Local Search (cont.)
Graph Equipartition Problem: Given a graph G = (V, E) and n = |V |, the problem is
to nd a subset S V with |S| =
_
n
2
_
for which the number c(S) of edges in the cut set
(S, V \ S) is minimized, where
(S, V \ S) = {(i, j) E : i S, j / S} .
In this example all feasible sets have the same size,
_
n
2
_
. A natural neighborhood of a feasible
set S V therefore consists of all subsets of nodes obtained by replacing one element in S
by one element not in S:
Q(S) = {T N : |T \ S| = |S \ T| = 1} .
3
4
2
5
1
6
Example: In the graph shown left we start with S = {1, 2, 3} for
which c(S) = 6. Then
Q(S) = {{1, 2, 4} , {1, 2, 5} , {1, 2, 6} , {1, 3, 4} , {1, 3, 5} ,
{1, 3, 6} , {2, 3, 4} , {2, 3, 5} , {2, 3, 6}}.
for which c(T) = 6, 5, 4, 4, 5, 6, 5, 2, 5 respectively. The new
incumbent is S = {2, 3, 5} with c(S) = 2. Q(S) does not
contain a better solution, as may be easily veried, so S = {2, 3, 5}
is locally optimal.
Optimization Group 18