MATHEMATICAL PROGRAMMING – 2

Sessions-2(CO1)

Topic-Convex Optimization, Conic

Programming, Linear Programming (LP)

Textbook/Article : Convex Optimization by Stephen Boyd and Lieven

Vandenberghe.
 Relationship between
different programming.

During the last two decades, major developments in Convex Optimization were focusing on Conic
Programming(CP), LP, Conic Quadratic and Semidefinite optimization.
Mathematical optimization

3
Solving optimization problem
 What is Linear Programming?
• Linear programming is a simple technique where we depict complex
relationships through linear functions and then find the optimum
points. The important word in the previous sentence is depicted. The
real relationships might be much more complex – but we can simplify
them to linear relationships.
• Example of a linear programming problem
• Let’s say a FedEx delivery man has 6 packages to deliver in a day. The
warehouse is located at point A. The 6 delivery destinations are given
by U, V, W, X, Y, and Z. The numbers on the lines indicate the
distance between the cities. To save on fuel and time the delivery
person wants to take the shortest route.
• So, the delivery person will calculate different routes for going to all
the 6 destinations and then come up with the shortest route. This
technique of choosing the shortest route is called linear
programming.

5
Cont…

In this case, the objective of the delivery person is to deliver the parcel on
time at all 6 destinations. The process of choosing the best route is called
Operation Research. Operation research is an approach to decision-
making, which involves a set of methods to operate a system. In the above
example, my system was the Delivery model.

6
Cont…
• Linear programming is used for obtaining the most optimal solution
for a problem with given constraints. In linear programming, we
formulate our real-life problem into a mathematical model. It involves
an objective function, linear inequalities with subject to constraints.
• Formulating a problem – Let’s manufacture some chocolates
• Example: Consider a chocolate manufacturing company that produces only two types
of chocolate – A and B. Both the chocolates require Milk and Choco only.  To
manufacture each unit of A and B, the following quantities are required:
• Each unit of A requires 1 unit of Milk and 3 units of Choco
• Each unit of B requires 1 unit of Milk and 2 units of Choco
• The company kitchen has a total of 5 units of Milk and 12 units of Choco. On each
sale, the company makes a profit of
• Rs 6 per unit A sold
• Rs 5 per unit B sold.
• Now, the company wishes to maximize its profit. How many units of A and B should it
produce respectively?

7
Cont…
Solution: The first thing is represent the problem in a tabular
form for better understanding.
 
Let the total number of units produced by A be = X
Let the total number of units produced by B be = Y
Now, the total profit is represented by Z
The total profit the company makes is given by the total number
of units of A and B produced multiplied by its per-unit profit of
Rs 6 and Rs 5 respectively.
Profit: Max Z = 6X+5Y
which means we have to maximize Z.
Profit per
Milk Choco
unit
A 1 3  Rs 6
B 1 2  Rs 5
Total 5 12

8
Cont…
• The company will try to produce as many units of A and B to maximize the
profit. But the resources Milk and Choco are available in a limited amount.
• As per the above table, each unit of A and B requires 1 unit of Milk. The total
amount of Milk available is 5 units. To represent this mathematically,
• X+Y ≤ 5
• Also, each unit of A and B requires 3 units & 2 units of Choco respectively.
The total amount of Choco available is 12 units. To represent this
mathematically,
• 3X+2Y ≤ 12
• Also, the values for units of A can only be integers.
• So we have two more constraints, X ≥ 0  &  Y ≥ 0
• For the company to make maximum profit, the above inequalities
have to be satisfied.
• This is called formulating a real-world problem into a mathematical
model.

9
 Common terminologies used in Linear
Programming
• Decision Variables: The decision variables are the variables that will decide my output.
They represent my ultimate solution. To solve any problem, we first need to identify
the decision variables. For the above example, the total number of units for A and B
denoted by X & Y respectively are my decision variables.
• Objective Function: It is defined as the objective of making decisions. In the above
example, the company wishes to increase the total profit represented by Z. So, profit is
my objective function.
• Constraints: The constraints are the restrictions or limitations on the decision
variables. They usually limit the value of the decision variables. In the above example,
the limit on the availability of resources Milk and Choco are my constraints.
• Non-negativity restriction: For all linear programs, the decision variables should always
take non-negative values. This means the values for decision variables should be
greater than or equal to 0.

10
Solve Linear Programs by Graphical Method
Example: A farmer has recently acquired a 110 hectares piece
of land. He has decided to grow Wheat and barley on that land.
Due to the quality of the sun and the region’s excellent climate,
the entire production of Wheat and Barley can be sold. He
wants to know how to plant each variety in the 110 hectares,
given the costs, net profits and labor requirements according to
the data shown below:
The farmer has a budget of US$10,000 and availability of 1,200
man-days during the planning horizon. Find the optimal solution
and the optimal value.

 Net
Cost  Man-
Profit
Variety (Price/H days/
(Price/H
ec) Hec
ec)
Wheat 100  50  10
Barley 200  120  30
11
Cont…
• Formulation of Linear Problem
• Step 1: Identify the decision variables
• The total area for growing Wheat = X (in hectares)
• The total area for growing Barley  = Y (in hectares)
• X and Y are my decision variables.
•  
• Step 2: Write the objective function
• Since the production from the entire land can be sold in the market. The farmer would want to maximize the profit for his
total produce. We are given net profit for both Wheat and Barley. The farmer earns a net profit of US$50 for each hectare of
Wheat and US$120 for each Barley.
• Our objective function (given by Z) is, Max Z = 50X + 120Y
•  
• Step 3: Writing the constraints 
• 1. It is given that the farmer has a total budget of US$10,000. The cost of producing Wheat and Barley per hectare is also
given to us. We have an upper cap on the total cost spent by the farmer. So our equation becomes:
• 100X + 200Y ≤ 10,000 
• 2. The next constraint is the upper cap on the availability of the total number of man-days for the planning horizon. The total
number of man-days available is 1200. As per the table, we are given the man-days per hectare for Wheat and Barley.
• 10X + 30Y ≤ 1200
• 3. The third constraint is the total area present for plantation. The total available area is 110 hectares. So the equation
becomes,
• X + Y ≤ 110

12
Cont…
• Step 4: The non-negativity restriction
• The values of X and Y will be greater than or equal to 0. This goes without saying.
• X ≥ 0, Y ≥ 0
• We have formulated our linear program. It’s time to solve it.
• Solving an LP through Graphical method
• Since we know that X, Y ≥ 0. We will consider only the first quadrant.
• To plot for the graph for the above equations, first I will simplify all the equations.
• 100X + 200Y ≤ 10,000 can be simplified to X + 2Y ≤ 100 by dividing by 100.
• 10X + 30Y ≤ 1200 can be simplified to X + 3Y ≤ 120 by dividing by 10.
• The third equation is in its simplified form, X + Y ≤ 110.
• Plot the first 2 lines on a graph in the first quadrant (like shown below)
• The optimal feasible solution is achieved at the point of intersection where the budget & man-days
constraints are active. This means the point at which the equations X + 2Y ≤ 100 and X + 3Y ≤ 120
intersect gives us the optimal solution.
• The values for X and Y which gives the optimal solution is at (60,20).
• To maximize profit the farmer should produce Wheat and Barley in 60 hectares and 20 hectares of
land respectively.
• The maximum profit the company will gain is,
• Max Z = 50 * (60) + 120 * (20)
• =  US$5400

13
Cont…

14
Simplex Method
Simplex Method is one of the most powerful & popular methods for linear
programming. The simplex method is an iterative procedure for getting the
most feasible solution. In this method, we keep transforming the value of
basic variables to get maximum value for the objective function.
A linear programming function is in its standard form if it seeks to maximize
the objective function. 

subject to constraints,

where, 

15
Cont…
After adding slack variables, the corresponding system of constraint equation
is,

where, 

are called slack variables. They are non-negative numbers that are added to
remove the inequalities from an equation.
The above explanation gives the theoretical explanation of the simplex
method. Now, I am gonna explain how to use the simplex method in real life
using Excel.

16
Convex optimization
• Convex optimization is a subfield of 
mathematical optimization that studies the problem of
minimizing convex functions over convex sets.
 Convex, Strictly convex, Concave, Strictly
concave

• A function f (x) is called convex if, for every y and z and

every 0 ≤ λ ≤ 1, f [λy + (1 − λ)z] ≤ λf (y) + (1 − λ) f (z).
• It is called strictly convex if, for every two distinct points y
and z and every 0 < λ < 1, f [λy + (1 − λ)z] < λf (y) + (1 −
λ) f (z).

• A function f (x) is called concave if,for every y and z and

every 0 ≤ λ ≤ 1, f [λy + (1 − λ)z] ≥ λf (y) + (1 − λ) f (z).
• It is called strictly concave if, for every y and z and every
0 < λ < 1,
f [λy + (1 − λ)z] > λf (y) + (1 − λ) f (z).
Cont…
• We can easily show that a linear function is both convex and
concave. Consider the linear function:

and let 0 ≤ λ ≤ 1.
Then
Convex cone:
A set C is said to be affine if for any two points x, y ∈ C and any real
numbers α and β, the affine combination point αx + βy ∈ C.
A set C is said to be convex if for any x, y ∈ C and any real number α, 0 < α
< 1, the convex combination point αx + (1 − α)y ∈ C.
The convex hull of a set Ω is the intersection of all convex sets containing Ω.
• Proposition 1.1 Let C1 and C2 be convex sets in a same space. Then,
• • C1 ∩ C2 is convex.
• • C1 + C2 is convex, where C1 + C2 = {b1 + b2 : b1 ∈ C1 and b2 ∈ C2}.
• • C1 ⊕ C2 is convex, where C1 ⊕ C2 = {(b1; b2) : b1 ∈ C1 and b2 ∈
C2}. Let us use the notation E to represent either Rn or S n, depending on
the context, throughout this book, because all our decision and
optimization problems take variables from one or both of these two vector
spaces. A set K ⊂ E is a cone if x ∈ K implies αx ∈ K for all α > 0. A cone
that is also convex is a convex cone. For a cone K ⊂ E, the dual of K is
the cone K∗ := {y : hx, yi ≥ 0 for all x ∈ K}, where again h·, ·i is the inner
product operation for space E

20
 Why convexity is the key to optimization

• It is easy with convex cost functions

• The most interesting thing you would first come across when
starting out with machine learning is the optimization
algorithm and to be specific, it is the gradient descent, which
is a first-order iterative optimization algorithm used to
minimize the cost function.
• The intuition behind gradient descent is converging to a
solution, which could be a local minimum in the neighborhood
or in best-case, the global minimum.
• Everything seems fine until you start questioning yourself
about the convergence problem. Having a good understanding
of convexity helps you prove the intuition behind the idea of
gradient descent.
21
Convex Sets
To simply things, think of convex sets as shapes where any line joining 2 points in
this set is never outside the set. This is called a convex set.

22
 Cont…
It is evident that any line joining 2 points on a circle or say a square (the shapes on extreme left and
middle), will have all the line segments within the shape. These are examples of convex sets.
On the other hand, the shape on the extreme right in the figure above has part of a line outside the
shape. Thus, this is not a convex set.
A convex set C can be represented as follows.

                                          
                                          

23
 Epigraph
Consider the graph of a function f.
An epigraph is a set of points lying on or above the function’s graph.

24
Convex function

Convex Function
A convex function with its epigraph. A function f is said to be a convex
function if its epigraph is a convex set (as seen in the green figure below
on the left).
This means that every line segment drawn on this graph is always equal to or
above the function graph. Pause a minute and check for yourself.
         
         

25
 Cont…
This means that a function f is not
convex if there exist two points x, y such
that the line segment joining f(x) and
f(y), is below the curve of the function
f. This causes the loss of convexity of the
epigraph (as seen in the red-figure above
on the right ).
This means that every line segment drawn
on this graph is not always equal to or
above the function graph. The same can
be proven by taking points on the bends.

26
 Testing for convexity
Most of the cost functions in the case of neural networks would be non-convex. Thus you must test a
function for convexity.
A function f is said to be a convex function if the seconder-order derivative of that function is greater
than or equal to 0.

          
Condition for convex functions.
Examples of convex functions: y=eˣ, y=x². Both of these functions are differentiable twice.
If -f(x) (minus f(x)) is a convex function, then the function is called a concave function.
                
                
Condition for concave functions.
Examples of concave functions: y=-eˣ. The function is differentiable twice.

27
THANK
YOU