Optimization class Presentation for students only

1. 1
Linear & Integer Programming : Introduction
Engg System Design Optimization
M S Prasad

2. General Optimization Program
• Standard form:
• where,
• Too general to solve, must specify properties
of X, f,g and h more precisely.

3. Complexity Analysis
• (P) – Deterministic Polynomial time algorithm
• (NP) – Non-deterministic Polynomial time
algorithm,
– Feasibility can be determined in polynomial time
• (NP-complete) – NP and at least as hard as
any known NP problem
• (NP-hard) – not provably NP and at least as
hard as any NP problem,
– Optimization over an NP-complete feasibility
problem

4. Optimization Problem Types –
Real Variables
• Linear Program (LP)
– (P) Easy, fast to solve, convex
• Non-Linear Program (NLP)
– (P) Convex problems easy to solve
– Non-convex problems harder, not guaranteed to find global
optimum

5. Let: X1, X2, X3, ………, Xn = decision variables
Z = Objective function or linear function
Requirement: Maximization of the linear function Z.
Z = c1X1 + c2X2 + c3X3 + ………+ cnXn …..Eq (1)
subject to the following constraints:
Problems

6. 6
Linear programming (LP) model
• LP:
• Matrix form:
)n,,1j(0x
bxa...xaxa
bxa...xaxa
bxa...xaxa
:tosubject
xc...xcxcmin
j
mnmn22m11m
2nn2222121
1nn1212111
nn2211


=≥
≤+++
≤+++
≤+++
+++
0x
bAx
xcmin T
≥
≤
objective function
constraints
variable
restrictions
where:
x, c: n-vector
A: m,n-matrix
b: m-vector

7. 7
Linear programming example
)2,1j(0x
12x4x3
4xx2
:tosubject
xxmax
j
21
21
21
=≥
≤+
≤+
+






≥











≤























0
0
x
x
12
4
x
x
43
12
:tosubject
x
x
1
1
max
2
1
2
1
2
1
T
or:

8. 8
Linear programming example:
graphical solution (2D)
1 2 3 4 5 6
x1
1
2
3
4
5
6x2
)objective(
6xx 21 =+
4xx2 21 ≤+
12x4x3 21 ≤+
)2,1j(0x
12x4x3
4xx2
:tosubject
xxmax
j
21
21
21
=≥
≤+
≤+
+
0
solution
space

9. 9
Linear programming (cont.)
• Decision variables can take the fractional
values. ( May not suitable in some areas such
as Resource mgmt , allocation etc)
• Solution techniques:
– (dual) simplex method
– interior point methods (e.g. Karmarkar algorithm)

10. 10
Integer programming (IP) models
• Integer variable restriction
– IP: integer variables only
– MIP: part integer, part non-integer variables
– BIP: binary (0-1) variables
• General IP-formulation:
• Complex solution space
)Zx(integerx
bAx
xcmin T
+
∈
≤

11. 11
Integer programming example:
graphical solution (2D)
1 2 3 4 5 6
x1
1
2
3
4
5
6x2
)objective(
6xx 21 =+
)2,1j(Zx
12x4x3
4xx2
:tosubject
xxmax
j
21
21
21
=∈
≤+
≤+
+
+
0
2 optimal solutions!

12. 12
Total unimodularity property for
integer programming models
Suppose that all coefficients are integer in the model:
i.e.
Example: transportation problem
if A has the total unimodularity property
(i.e. every square submatrix has determinant 0,1,-1)
⇒ there is an optimal integer solution x*
& the simplex method will find such a solution
0x
bAx
xcmin T
≥
≤ j,i,b,a iij ∀Ν∈

13. 13
Integer programming tricks
PROBLEM: x = 0 or x ≥ k
use binary indicator variable y=
restrictions:
kfor x,
0for x,
1
0
≥
=



{ }0,1y
ykx
x)onupperboundanis(MyMx
∈
⋅≥
⋅≤

14. 14
PROBLEM: fixed costs: if xi>0 then costs C(xi)
use indicator variable yi=
restrictions :
Integer programming tricks (2)
0x
bAx
)x(Cminimize
≥
=
0.for x
0,for x
xck
0
)x(C
:where
i
i
iii
i
>
=



+
=
0for x,
0for x,
1
0
i
i
>
=



{ }0,1y
xcyk)x(C
yMx
i
iiiii
ii
∈
⋅+⋅=
⋅≤
)i(∀

15. 15
• Hard vs. soft restrictions
– hard restriction: must hold, otherwise unfeasibility
for example:
– soft restriction: may be violated, with a penalty
for example:
(Integer) programming tricks (3)
0Y,0x
Y5xx
100Yxcminimize
21
T
≥≥
+≤+
⋅+

5xx 21 ≤+

16. 16
• Absolute values:
solution:
(Integer) programming tricks (4)
−+
−+
+=⇒
−=
ttt
ttt
yyy
yyy
freey,0x
ybxa
ymin
tt,j
ttt,j
j
j
j
t
≥
+=∑
∑
( )
0y,0y,0x
yybxa
yymin
ttt,j
tttt,j
j
j
t
tt
≥≥≥
−+=
+
−+
−+
−+
∑
∑
goal
variation

17. 17
• Conjunctive/disjunctive programming
- conjunctive set of constraints: must all be satisfied
- disjunctive set of constraints: at least one must be satisfied
• example (Appendix A.4):
Integer programming tricks (5)

jkj
kjk
j
jj
pxx
or
pxx
xwmin
≥−
≥−
∑
}1,0{y
)y1(Mpxx
yMpxx
xwmin
2jkj
1kjk
j
jj
∈
−⋅−≥−
⋅−≥−
∑


18. 18
IP example
nonpreemptive single machine, total weighted
completion time (App. A.3)
function)(objectivextwimizemin
jjoboftimecompletionxt
jjoboftimecompletionxt
n
1j
1-Cmax
0t
jtj
1-Cmax
0t
jt
jt
∑ ∑
∑
= =
=
⋅⋅⇒
=⋅⇒
==⋅⇒ 1xif jt
otherwise0andt,timeatjjobif,1xjt completes=
objective function: minimize weighted completion time:
model definition:

19. 19
IP example (cont.)
Restriction: all jobs must be completed once:
1x
1-Cmax
0t
jt =∑=
Restriction: only one job per time t:
t)timeperjoboneexactly:on(restricti1x
t)duringprocessinisjjob(if1x
n
1j
pt
ts
js
pt
ts
js
j
j
=⇒
=⇒
∑∑
∑
=
+
=
+
=
if job j is in process during t, it must be completed
somewhere during [t,t+pj]

20. 20
IP example (cont.)
Complete IP-model:
{ } 1)-Cmax,0,tn,,1,j(for1,0x
1)-Cmax,0,t(for1x
n),1,j(for1x
:tosubject
xtwinimizem
jt
n
1j
pt
ts
js
1-Cmax
0t
jt
n
1j
1-Cmax
0t
jtj
j



==∈
==
==
⋅⋅
∑∑
∑
∑ ∑
=
+
=
=
= =
n×Cmax
integer
variables

21. 21
IP example (cont.)
Additional restriction: precedence constraints
Model definition: SUCC(j) = successors of job j
⇒ job j must be completed before all jobs in SUCC(j):
n),1,jSUCC(j),k(fortxtx
kjoboftimestartptx
jjoboftimecompletiontx
1maxC
0t
kt
1maxC
0t
jt
k
1maxC
0t
kt
1maxC
0t
jt
=∈≤⇒
=−
=
∑∑
∑
∑
−
=
−
=
−
=
−
=

22. 22
Integer programming
solution techniques
• Heuristic vs. explicit approach:
– trade-off between solution quality and computation time
– trade-off between implementation effort/costs and yield (i.e.
profits gained from solution quality improvement)
• Heuristic methods; for example:
– local search (e.g. simulated annealing, tabu search, k-opt)
– (composite) dispatching rules (e.g. EDD, SPT, MS)
– adaptive search
– rounding fractional solutions
– beam search

23. 23
• Explicit methods; 3 categories:
1. dynamic programming
2. cutting plane (polyhedral) methods
3. branch and bound
or: hybrid methods (combination of the above)
• Commercial IP solvers usually use a
combination of heuristics and 2, 3
Integer programming
solution techniques (cont.)

24. 24
Dynamic programming
• Problem divided into stages
• Each stage can have various states
• A recursive objective function is used to
iterate through all states and all stages
(forwards or backwards)
))}x,i(i(F)x,i(c{min)i(F tt1t1tttt
x
tt
t
−−+=
T),0,(txt =
ti
(constant)c)i(F 000 =

25. 25
Cutting plane methods
STEP 0: Create a relaxation of the problem by
omitting restrictions
(e.g. the integrality restrictions)
STEP 1: Solve the current problem
STEP 2: If solution is infeasible then generate
a restriction that cuts of the solution,
and add it to the problem ⇒ STEP 1
Otherwise: DONE

26. 26
Branch and bound
• Enumeration in a search tree
• each node is a partial solution, i.e. a part of
...
...
root node
child nodes
child nodes
Level 0
Level 1
Level 2

27. 27
Branch and bound example 1
Disjunctive programming (appendix A.4):
disjunctive set of constraints: at least one must be satisfied
xj = completion time of job j
restriction: )Ik,j(pxxorpxxeither jkjkjk ∈∀≥−≥−
solve LP without
disjunctive restrictions
(= LP relaxation)
if disjunct. restr.
violated for j & k
Level 0
Level 1
...
kjk pxx ≥− jkj pxx ≥−

28. 28
Branch and bound (cont.)
• Upper bound: e.g. a feasible solution
• Lower bound:
e.g. a solution to an “easier” problem
• Node elimination (fathom/discard nodes):
when lower bound >= upper bound

29. 29
Branch and bound (cont.)
• Branching strategy:
how to partition solution space
• Node selection strategy:
– sequence of exploring nodes:
• depth first (tries to obtain a solution fast)
• breadth/best bound first (tries to find the best solution)
– which nodes to explore (filter and beam width)
• filter width: #nodes selected for thorough evaluation
• beam width: #nodes that are branched on (≤ filter width)
⇒ Beam search

30. 30
Branch and bound example 2
• Single machine, maximum lateness, release
and due dates
lower bound: EDD + preemption
Jobs 1 2 3 4
p(j) 4 2 6 5
r(j) 0 1 3 5
d(j) 8 12 11 10
(?,?,?,?)
(1,?,?,?) (4,?,?,?)(2,?,?,?)
Level 0
Level 1(3,?,?,?)

31. 31
Branch and bound example 2
• Lower bound for: (1,?,?,?)
Lower bound: Lmax = max(0,17-12,15-11,0)=5
Jobs 1 2 3 4
p(j) 4 2 6 5
r(j) 0 1 3 5
d(j) 8 12 11 10
t
r(2) r(3) r(4)
d(4)<d(3) d(3)<d(2)
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
3 3 21 4

32. 32
(?,?,?,?)
(1,?,?,?) (4,?,?,?)(2,?,?,?)
Level 0
Level 1(3,?,?,?)
Branch and bound example 2 (cont.)
LB=5 LB=7*
=UB
(1,2,?,?) (1,3,?,?) Jobs 1 2 3 4
p(j) 4 2 6 5
r(j) 0 1 3 5
d(j) 8 12 11 10
infeasible:
(1,3,4,3,2)
LB=6*
=UB
(1,2,4,3)
LB=5*=UB
(1,3,4,2)
DONE
(1,2,4,3) (1,3,4,2)

33. 33
Branch and bound example 3
1 2 3 4 5 6
x1
1
2
3
4
5
6x2
)objective(
6xx 21 =+
0
4.2x
,8.0x
2
1
=
=
0x1 ≤ 1x1 ≥
3x
,0x
2
1
=
=
2x
,1x
2
1
=
=
2x2 ≤ 3x2 ≥
2x
,1x
2
1
=
=
3x
,0x
2
1
=
=
obj: 3 obj: 3
obj: 3 obj: 3
LP solution:

34. 34
Beam search example 1
single-machine, total weighted tardiness
Upper bound: ATC rule (apparent tardiness cost):
• schedule 1 job at a time
• every time a machine comes available, determine
ranking of jobs:








⋅
−−
−
=
pK
)0,tpdmax(
j
j
j
jj
e
p
w
)t(I
MS rule
WSPT
rule
look-ahead parameter:
K = 4.5 + R (R ≤ 0.5)
K = 6 - 2R (R ≥ 0.5)
= due date range factor( ) maxminmax C/ddR −=

35. 35
Beam search example 1 (cont.)
single-machine, total weighted tardiness
(?,?,?,?)
(1,?,?,?) (4,?,?,?)(2,?,?,?) (3,?,?,?)
1,2,3)(j
p
w
e
p
w
)t(I
j
jpK
)0,tpdmax(
j
j
j
jj
===⇒








⋅
−−
−
Upper bound by ATC rule: )3,2,1j(0)0,tpdmax( jj ==−−
Jobs 1 2 3 4
p(j) 10 10 13 4
d(j) 4 2 1 12
w(j) 14 12 1 12
w(j)/p(j) 1.4 1.2 0.1 3

36. 36
Beam search example 1 (cont.)
single-machine, total weighted tardiness
(?,?,?,?)
(1,?,?,?) (4,?,?,?)(2,?,?,?) (3,?,?,?)
Jobs 1 2 3 4
p(j) 10 10 13 4
d(j) 4 2 1 12
w(j) 14 12 1 12
w(j)/p(j) 1.4 1.2 0.1 3
Jobs C(j) d(j) T(j) w(j)*T(j)
1 10 4 6 84
2 24 2 22 264
3 37 1 36 36
4 14 12 2 24
Upper bound
by ATC rule:
Total = 408

37. 37
Beam search example 1 (cont.)
single-machine, total weighted tardiness
(?,?,?,?)
(1,?,?,?) (4,?,?,?)(2,?,?,?) (3,?,?,?)
UB=408 UB=436 UB=814 UB=440
discardedexplored
further
(beam width = 2)
4 nodes
analyzed
(filter width=4)

38. 38
Beam search example 1 (cont.)
(?,?,?,?)
(1,?,?,?) (4,?,?,?)(2,?,?,?) (3,?,?,?)
UB=408 436 814 440
(1,2,?,?) (1,3,?,?) (1,4,?,?)
UB=480 706 408
(1,4,2,3) (1,4,3,2)
UB=408 554
best solution
(2,1,?,?) (2,3,?,?) (2,4,?,?)
436
(2,4,1,3) (2,4,3,1)
436 608