This document discusses various methods for solving unconstrained multidimensional optimization problems of the form minx∈Rnf(x). It begins by introducing the necessary condition that the gradient of f must be zero at the minimum x* and the sufficient condition that the Hessian must be positive definite. Both direct and indirect methods are then described. Direct methods include univariate search and the simplex method, while indirect methods involve choosing a search direction and step size. Specific indirect methods covered in detail include steepest descent, Newton's method, quasi-Newton methods like DFP and BFGS, and the conjugate gradient method.
This document discusses various methods for solving unconstrained multidimensional optimization problems of the form minx∈Rnf(x). It begins by introducing the necessary condition that the gradient of f must be zero at the minimum x* and the sufficient condition that the Hessian must be positive definite. Both direct and indirect methods are then described. Direct methods include univariate search and the simplex method, while indirect methods involve choosing a search direction and step size. Specific indirect methods covered in detail include steepest descent, Newton's method, quasi-Newton methods like DFP and BFGS, and the conjugate gradient method.
This document discusses various methods for solving unconstrained multidimensional optimization problems of the form minx∈Rnf(x). It begins by introducing the necessary condition that the gradient of f must be zero at the minimum x* and the sufficient condition that the Hessian must be positive definite. Both direct and indirect methods are then described. Direct methods include univariate search and the simplex method, while indirect methods involve choosing a search direction and step size. Specific indirect methods covered in detail include steepest descent, Newton's method, quasi-Newton methods like DFP and BFGS, and the conjugate gradient method.
This document discusses various methods for solving unconstrained multidimensional optimization problems of the form minx∈Rnf(x). It begins by introducing the necessary condition that the gradient of f must be zero at the minimum x* and the sufficient condition that the Hessian must be positive definite. Both direct and indirect methods are then described. Direct methods include univariate search and the simplex method, while indirect methods involve choosing a search direction and step size. Specific indirect methods covered in detail include steepest descent, Newton's method, quasi-Newton methods like DFP and BFGS, and the conjugate gradient method.
Sebastian Engell Process Dynamics and Operations Group
4. Multidimensional Optimization 4.1 Unconstrained Problems min xR
f(x)
f (x ) f(x*)
x2
Taylor series expansion of f(x):
( )+ f (x ) (x x )
f (x ) f x
(0 )
(0 )
x 1 ( 0) Gradient: f x = f x n
(0 )
( )
Process Optimization
( )(
x*
1 (0 ) T (0 ) (0 ) + xx H x x x + 2
2 2f
2 x1xn x1
Hessian: H x (0 ) = 2
2 f
2 xn x1 x
x (0 )
( )
x (0 )
x1
20114.1.1
Professor Dr. Sebastian Engell
Process Dynamics and Operations Group
Analytical Solution
( )
necessary condition for a minimum: f x * = 0
sufficient condition:
( )
H x*
is positive definite
(all eigenvalues are positive)
The analytical solution is only possible, if:
f(x) is twice differentiable,
( )
* the set of (nonlinear) equations f x = 0 can be solved for x*.
otherwise: use numerical optimization methods
(again: direct and indirect methods) Process Optimization
20114.1.2
Professor Dr. Sebastian Engell
Process Dynamics and Operations Group
Direct Methods Univariate search: fix n-1 variables, search for the optimum of the remaining; alternate between the coordinates
Simplex Method:
evaluate f for the vertices of a
simplex; reflect the vertex with the maximum f-value
Alternative Methods:
x*
x2
x(1)
x(0) x2
x1
x*
reduce the triangle size
when x* is contained
x1
Nelder-Mead method: simplex with expanding/ contracting polyhedra
Grid-based approaches Random Search Method of conjugate directions (Powells method)
[Textbook by Edgar, Himmelblau, and Lasdon: Optimization of Chemical Processes]
Process Optimization
20114.1.3
Professor Dr. Sebastian Engell
Process Dynamics and Operations Group
Indirect Methods Abstract scheme: (a) choose an initial point x(0); step index: i = 0 (b) determine a search direction d(i) s(i):
(c) determine a stepsize
either just choose a constant s(i), or by line search in direction d(i):
min f x (i ) + s (i ) d (i ) s >0
(i +1) = x (i ) + s (i ) d (i ) result: x
(d) stop if
f x (i +1) <
x(1) x2
x(2) s(0)
x*
d(0) x(0)
x1
to go in direction of decreasing cost:
( )
( )
( i) T f x (i ) < 0 d
else go to (b) with i = i + 1
Process Optimization
20114.1.4
Professor Dr. Sebastian Engell
Process Dynamics and Operations Group
Steepest Descent Method
Determination of the search direction: f = constant
opposite to the gradient:
( )
d (i ) = f x (i )
d(i) s(i)
(orthogonal to lines of equal cost)
+ converges quickly to a neighborhood of x*
+ robust with respect to the choice of x(0) - slow convergence near to x* (since f 0)
( ) *
- the result must be checked ( H x > 0 )
Process Optimization
20114.1.5
Professor Dr. Sebastian Engell
Process Dynamics and Operations Group
Newton Method as in the scalar case: Taylor series approximation of f (x) and use of the necessary condition
with stepsize:
( ) ( ) x (i +1) = x (i ) s (i ) H 1(x (i ) ) f (x (i ) ) x (i +1) = x (i ) H 1 x (i ) f x (i )
search direction d(i)
The cost decreases in direction d(i), if H(x(i)) is positive definite.
+ quadratic convergence - sensitive with respect to the choice of x(0) - high computational costs due to matrix inversion in each step Process Optimization
20114.1.6
Professor Dr. Sebastian Engell
Process Dynamics and Operations Group
Quasi-Newton Method (1)
Idea: reduce the costs for determination of d(i) by using symmetric approximations G(i) of H-1(x(i))
Scheme:
( ) (0 ) y (i 1) = f (x (i ) ) f (x (i 1) ) , G = I
x (i +1) = x (i ) + s (i ) d (i ) with d (i ) = G f x (i )
and G(i) from: (i 1) = x (i ) x (i 1) ,
DFP (Davidson, Fletcher, Powell) formula:
T
T G y y G ( i) T G = G + T
y y G y
Process Optimization
(i 1)
20114.1.7
Professor Dr. Sebastian Engell
Process Dynamics and Operations Group
Quasi-Newton Method (2)
BFGS (Broyden, Fletcher, Goldfarb, Shano) formula: T T T T
y G y y G G y
G (i ) = G + 1 + T
T T
y y y
(i 1)
Properties: + convergence between linear and quadratic + smaller effort as for the Newton method
Process Optimization
20114.1.8
Professor Dr. Sebastian Engell
Process Dynamics and Operations Group
Conjugate Gradient Method
idea: choose linear independent and conjugate directions,
( i 1) T i.e.,: d Q d (i ) = 0
with a positive definite Q
f T (i ) f (i ) ( ( 0) 0) ( ( ( i) i) i 1) scheme: d = f x , d = f x + d T (i 1) f f (i 1)
( )
( )
+ good convergence; effort slightly higher as for steepest descent but
much more efficient
Line Search Step: min f x (i ) + s (i ) d (i )
s >0
) )
numerical solution of: f T x (i ) + s (i ) d (i ) d (i ) = 0
