Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Derivative Free Optimization and Robust Optimization


Published on

AACIMP 2009 Summer School lecture by Gerhard Wilhelm Weber. "Modern Operational Research and Its Mathematical Methods" course.

Published in: Education, Technology
  • Login to see the comments

Derivative Free Optimization and Robust Optimization

  1. 1. 4th International Summer School Achievements and Applications of Contemporary Informatics, Mathematics and Physics National University of Technology of the Ukraine Kiev, Ukraine, August 5-16, 2009 Nonsmooth Optimization Derivative Free Optimization and Robust Optimization Gerhard-Wilhelm Weber * and Başak Akteke-Öztürk Gerhard- Akteke- Institute of Applied Mathematics Middle East Technical University, Ankara, Turkey * Faculty of Economics, Management and Law, University of Siegen, Germany Center for Research on Optimization and Control, University of Aveiro, Portugal
  2. 2. Introduction Mathematical Models • Experimental Data Analysis • Classification problems • Identification problems treated by: • Pattern Recognition SVM, Cluster Analysis, • Assignment and Allocation Neural Systems etc. • When these methods were born, the most developed and popular optimization tools were Linear and Quadratic Programming. • Optimization parts of these methods are reduced to LP and QP Linear Discriminant Analysis
  3. 3. Introduction • progress in Optimization • new advanced tools, • Nonsmooth Analysis and • construct a mathematical model Nondifferentiable Opt. better suited for the problem under consideration • Most cases clustering problems are reduced to solving nonsmooth optimization problems. • We are interested in new methods for solving related nonsmooth problems (e.g., Semidefinite Programming, Semi-Infinite Programming, discrete gradient method and cutting angle method).
  4. 4. Nonsmooth Optimization Problem: minimize subject to • : is nonsmooth at many points of interest do not have a conventional derivative at these points. • A less restrictive class of assumptions for than smoothness: convexity and Lipschitzness.
  5. 5. Nonsmooth Functions
  6. 6. Convex Sets A set is called convex if
  7. 7. Convex Sets • The convex hull of a set : • The sets and coincide if and only if is convex. • The set is called a cone if for all , ; i.e., contains all positive multiples of its elements.
  8. 8. Convex hull
  9. 9. Convex Functions • A set is called an epigraph of function . • Let be a convex set. A function is said to be convex if its epigraph is a convex set.
  10. 10. Convex Functions • are differentiable (smooth) almost everywhere, • their minimizers are points where the function need not be differentiable, • standard numerical methods do not work • Examples of convex functions: – affinely linear: – quadratic: (c>0) – exponential:
  11. 11. Convex Functions
  12. 12. Convex Optimization • minimizing a convex function over a convex feasible set • Many applications. • Important, because: a strong duality theory any local minimum is a global minimum includes least-squares problems and linear programs as special cases can be solved efficiently and reliably
  13. 13. Lipschitz Continuous • A function is called (locally) Lipschitz continuous, if for any bounded there exist a constant such that • Lipschitzness is a more restrictive property on functions than continuity, i.e., all Lipschitz functions are continuous, but they are not guaranteed to be smooth. • They possess a generalized gradient.
  14. 14. Lipschitz Continuous
  15. 15. Nonsmooth Optimization • We call the the set ∂f(x) subdifferential of f at x • Any vector v є ∂f(x) is a subgradient. • A proper convex function f is subdifferentiable at any point x є , if ∂f(x) is non-empty, convex and compact at x. • If the convex function f is continuously differentiable, then
  16. 16. Nonsmooth Functions and Subdifferentials
  17. 17. Generalized Derivatives • The generalized directional derivative of f at x in the direction g is defined as • If the function f is locally Lipschitz continuous, then the generalized directional derivative exists. • The set is called the (Clarke) subdifferential of the function f at a point
  18. 18. Nonsmooth Optimization Nonsmooth optimization – more general problem of minimizing functions, – lack some, but not all, of the favorable properties of convex functions, – minimizers often are again points where the function is nondifferentiable.
  19. 19. Cluster Analysis via Nonsmooth Opt. Given Problem: This is a partitioning clustering problem.
  20. 20. Clustering
  21. 21. Clustering
  22. 22. Cluster Analysis via Nonsmooth Opt. • k is the number of clusters (given), • m is the number of available patterns (given), • is the j-th cluster’s center (to be found), • association weight of pattern , cluster j (to be found): • ( ) is an matrix, • objective function has many local minima.
  23. 23. Cluster Analysis via Nonsmooth Opt. Suggestion (if k is not given a priori): • Start from a small enough number of clusters k and gradually increase the number of clusters for the analysis until a certain stopping criteria met. • This means: If the solution of the corresponding optimization problem is not satisfactory, the decision maker needs to consider a problem with k + 1 clusters, etc.. • This implies: One needs to solve repeatedly arising optimization problems with different values of k - a task even more challenging. • In order to avoid this difficulty, we suggest a step-by-step calculation of clusters.
  24. 24. Cluster Analysis via Nonsmooth Opt. • k-means, h-means, j-means • dynamic programming • branch and bound • cutting planes • metaheuristics: simulated annealing, tabu search and genetic algorithms • an interior point method for minimum sum-of squares clustering problem • agglomerative and divisive hierarchical clustering incremental approach
  25. 25. Cluster Analsysis via Nonsmooth Opt. Reformulated Problem: • A very complicated objective function: nonsmooth and nonconvex. • The number of variables in the nonsmooth optimization approach is k×n, before it was (m+n)×k.
  26. 26. Robust Optimization • There is uncertainty or variation in the objective and constraint functions, due to parameters or factors that are either beyond our control or unknown. • Refers to the ability of the subject to cope well with uncertainties in linear, conic and semidefinite programming . • Applications in control, engineering design and finance. • Convex, modelled by SDP or cone quadratic programming. • Robust solutions are computed in polynomial time, via (convex) semidefinite programming problem.
  27. 27. Robust Optimization • Let us examine Robust Linear Programming • By a worst case approach the objective is the maximum over all possible realizations of the objective • A robust feasible solution with the smallest possible value of the f(x) is sought. • Robust optimization is no longer a linear programming. The problem depends on the geometry of the uncertainty set U; i.e., if U is defined as an ellipsoid, the problem becomes a conic quadratic program.
  28. 28. Robust Optimization
  29. 29. Robust Optimization • Considers that the uncertain parameter c belongs to a bounded, convex, uncertainty set • Stochastic Optimization: expected values, parameter vector u is modeled as a random variable with known distribution Robust Counterpart • Worst Case Optimization: the robust solution is the one that has the best worst case, i.e., it solves
  30. 30. Robust Optimization • A complementary alternative to stochastic programming. • Seeks a solution that will have a “good” performance under many/most/all possible realizations of the uncertain input parameters. • Unlike stochastic programming, no distribution assumptions on uncertain parameters – each possible value equally important (this can be good or bad) • Represents a conservative viewpoint when it is worst-case oriented.
  31. 31. Robust Optimization • Especially useful when – some of the problem parameters are estimates and carry estimation risk, – there are constraints with uncertain parameters that must be satisfied regardless of the values of these parameters, – the objective functions / optimal solutions are particularly sensitive to perturbations, – decision-maker can not afford low-probability high-magnitude risks.
  32. 32. Derivative Free Optimization The problem is to minimize a nonlinear function of several variables • the derivatives (sometimes even the values) of this function are not available, • arise in modern physical, chemical and econometric measurements and in engineering applications, • computer simulation is employed for the evaluation of the function values. The methods are known as derivative free methods (DFO).
  33. 33. Derivative Free Optimization Problem: • cannot be computed or just does not exist for every x , • is an arbitrary subset of , • is called the easy constraint, • the functions represent difficult constraints.
  34. 34. Derivative Free Optimization Derivative free methods • build a linear or quadratic model of the objective function, • apply a trust-region or a line-search to optimize the model; derivative based methods use a Taylor polynomial -based model; DFO methods use interpolation, regression or other sample-based models.
  35. 35. Derivative Free Optimization Six iterations of a trust-region algorithm.
  36. 36. Semidefinite Programming • Optimization problems where the variable is not a vector but a symmetric matrix which is required to be positive semidefinite. • Linear Programming Semidefinite Programming vector of variables a symmetric matrix nonnegativity constraint a positive semidefinite constraint • SDP is convex, has a duality theory and can be solved by interior point methods.
  37. 37. SVC via Semidefinite Programming • I try to reformulate the support vector clustering problem as a convex integer program and then relax it to a soft clustering formulation which can be feasibly solved by a 0-1 semidefinite program. • In the literature, k-means and clustering methods which use a graph cut model are reformulated as a semidefinite program and solved by using semidefinite programming relaxations.
  38. 38. Some References 1. Aharon Ben-Tal and Arkadi Nemirovski, Robust optimization methodology and applications. 2. Adil Bagirov, Nonsmooth optimization approaches in data Classification. 3. Adil Bagirov, Derivative-free nonsmooth optimization and its applications. 4. A. M. Bagirov, A. M. Rubinov, N.V. Soukhoroukova and J. Yearwood, Unsupervised and supervised data classification via nonsmooth and global optimization. 5. Laurent El Ghaoui, Robust Optimization and Applications. 6. Başak A. Öztürk, Derivative Free Optimization methods: Application in Stirrer Configuration and Data Clustering.
  39. 39. Thank you very much! Questions, please?