One of the primary drawbacks plaguing wider acceptance of surrogate models is their low fidelity (in general), which can be in a large part attributed to the lack of quantitative guidelines regarding the suitability of different models for diverse classes of problems. In this context, model selection techniques are immensely helpful in ensuring the selection and use of an optimal model for a particular design problem. A novel model selection technique was recently developed to perform optimal model search at three levels: (i) optimal model type (e.g., RBF), (ii) optimal kernel type (e.g., multiquadric), and (iii) optimal values of hyper-parameters (e.g., shape parameter) that are conventionally kept constant. The maximum and the median error measures to be minimized in this optimal model selection process are given by the REES error metrics, which have been shown to be significantly more accurate than typical cross-validation-based error metrics. Motivated by the promising results given by REES-based model selection, in this paper we develop a framework called Collaborative Surrogate Model Selection (COSMOS). The primary goal of COSMOS is to allow the selection and usage of globally competitive surrogate models. More specifically, this framework will offer an open online platform where users from within and beyond the Engineering Design (and MDO) community can submit training data to identify best surrogates for their problem, as well as contribute new and advanced surrogate models to the pool of models in this framework. This first-of-its-kind global platform will facilitate sharing of ideas in the area of surrogate modeling, benchmarking of existing surrogates, validation of new surrogates, and identification of the right surrogate for the right problem. In developing this framework, this paper makes three important fundamental advancements to the original REES-based model selection - (i) The optimization approach is modified through binary coding to allow surrogates with differing numbers of candidate kernels and kernels with differ- ing numbers of hyper-parameters. (ii) A robustness criterion, based on the variance of errors, is added to the existing criteria for model selection. (iii) Users are allowed to perform model selection for a specified region of the input space and not only for the entire input domain, subject to empirical constraints that are related to the relative sample strength of the region. The effectiveness of the COSMOS framework is demonstrated using a comprehensive pool of five major surrogate model-types (with up to five constitutive kernel types), which are tested on two standard test problems and an airfoil design problem.

1. Concurrent Surrogate Model Selection (COSMOS)
based on
Predictive Estimation of Model Fidelity
Souma Chowdhury#, Ali Mehmani*, and Achille Messac#
* Syracuse University, Department of Mechanical and Aerospace Engineering
# Mississippi State University, Bagley College of Engineering
The ASME International Design Engineering Technical Conferences (IDETC)
August 17 – 20, 2014, Buffalo, NY

2. Surrogate Modeling
Surrogate models are commonly used for providing a tractable and inexpensive
approximation of the actual system behavior, as an alternative
 To expensive computational simulations (e.g., CFD), or
 To the lack of a physical model in the case of experiment-derived data (e.g., testing
of new metallic alloys).
풘풊 흍( 풙 − 풙풊 )
2
Model Type Kriging RBF SVR . . .
Linear Exponential Gaussian Cubic Multiquadric . . .
Kernel/Basis function
Correlation parameter Shape parameter . . .
Hyper-Parameter value
풇 풙 =
풏
풊=ퟏ
흍 풓 = (풓ퟐ + 풄ퟐ) ퟏ/ퟐ
풓= 풙 − 풙풊
풄풍풐풘풆풓 < 풄 < 풄풖풑풑풆풓

3. Outline
• Background and Literature
• Research Objectives
• COSMOS Framework
• Predictive Estimation of Model Fidelity (PEMF)
• Numerical Experiments: Results
• Concluding Remarks
3

4. Surrogate Model Selection
4
 Intuitive model selection (experience-based selection)
Model selection based on an understanding of the data characteristics
and/or the application constraints.
• Development of general guidelines likely not practical due to problem diversity.
• A few candidate surrogates are generally considered.
• In MDO problems, characteristics of disciplinary phenomena may not be evident.
 Automated model selection
Model selection based on the quantitative decision-making
techniques. Automated selection can be performed at these levels:

5. Automated Model or Kernel Selection
5
 Error measures are used to select the model type and basis functions*
퐹∗ = argmin
퐹 ∈푭
휺( 푭)
best surrogate model
surrogate model error
set of candidate surrogates
 Popular error measures used for model selection include: (i) split sample,
(ii) cross-validation, (iii) bootstrapping, (iv) Schwarz’s Bayesian information
criterion (BIC), and (v) Akaike’s information criterion (AIC)
Method Model Type Selection Kernel Type Selection
Holena et al., 2011 
Jin et al., 2001 
Gano et al., 2006 
Chen et al., 2004 
Viana et al., 2009  

6. Hyper-parameter Optimization
To mitigate the possibility of constructing a suboptimal surrogate model for a
given Kernel function, one must perform hyper-parameter optimization.
• Martin et al. (AIAAJ, 2005) used MLE and cross-validation methods to find the optimum
6
hyper-parameter value for the Gaussian correlation function in Kriging.
• Mongillo et al. (SIAM, 2011) used MLE and leave-one-out cross-validation methods to select
an optimal shape parameter in a Gaussian RBF.
• Gorissen et al. (JMLR, 2009) used the leave-one-out cross-validation and AIC error measures
in the SUMO Toolbox to select the hyper parameter value(s) through a genetic algorithm.
Shape parameter, σ
RMSE
X
F
Branin-Hoo function:
RBF Multiquadric
model with different
HP values

7. Research Objectives
 The original PEMF-based surrogate model selection method performed
selection at all three levels based on the median and maximum error.
 Models with similar number of kernel choices and kernels with a single
hyper-parameter was considered.
 The objectives of this research is to advance the PEMF-based COSMOS:
1. By introducing additional selection criteria: (i) the variance of the surrogate error and
(ii) the predicted error at a greater number of sample points.
2. By modifying the optimization formulation to allow competition among surrogates with
differing numbers of candidate kernels, and kernels with differing numbers of HPs.
3. By testing the COSMOS framework with a comprehensive set of model types and
constitutive kernel types − 16 surrogate-kernel combinations with 0 to 2 HPs.
PEMF: Predictive Estimation of Model Fidelity (Mehmani et al., AIAA Scitech 2014) 7

8. COSMOS Framework
8
Pareto Filter
Generally, any two
selection criteria, based on
user-preference, could be
considered simultaneously

9. COSMOS: MATLAB-based GUI
COSMOS MATLAB-based GUI: Courtesy of Ali Mehmani 9

10. COSMOS: Optimization Formulation
 Separate MINLPs are run in parallel for
each HP class (defined by #HPs involved)
 All hyper-parameters (CHP) are scaled to the
range 0 to 1.
 The candidate model-kernel combinations
are integer-coded.
 A single integer variable (TSK) now identifies
the model-kernel type.
 NSGA-II is used to solve the MINLP
problems. 10
Hyper-Parameter
Values
Candidate
Model-Kernel
Combinations
Branin Hoo
Function

11. Surrogate Model Candidates
11

12. Predictive Estimation of Model Fidelity (PEMF)
The PEMF method is derived from the hypothesis that the accuracy of
approximation models is related to the amount of data resources
leveraged to train the model.
 PEMF can be perceived as a novel sequential implementation of k-fold
12
cross-validation, with carefully constructed error measures.
 The PEMF method analyzes the variation of the model error distribution
with increasing number of training points.
 The PEMF method is a model independent approach for surrogate error
quantification, and does not require any additional test points.
 The PEMF method has been shown to be 1-2 orders of magnitude more
accurate in error quantification compared to leave-one-out cross validation.
Mehmani et al., AIAA SDM 2013, AIAA Scitech 2014, and Aviation, SMO 2014

13. PEMF: Approach
13
Median Error Maximum Error
Using Lognormal distribution at every iteration:
Variation of the modal value of the
median/maximum error is represented by
퐸 = 푎푛푏 or 퐸 = 푎푒푏푛
Final
Surrogate
Final
Surrogate

14. PEMF Input-Output
14
Error Metrics Model type
I/O Training data
Kernel type HP values
All error values expressed in terms
of relative absolute errors.

15. Numerical Experiments
Problem
Problem Settings Optimization Settings
Dimension Sample Size Population Size Max. Generations
Branin Hoo 2 30 20 50
Hartman-6 6 60 40 50
Dixon & Price 30 60 30 30
Neumaier & Perm 20 50 30 30
Airfoil Design 4 30 20 50
15
 Case 1: Minimize modal value of the median error and modal value of
the maximum error.
 Case 2: Minimize modal value of the median error and variance of the
median error.
 Case 3: Minimize modal value of the median error and the expected
modal value of the median error at 20% more sample points.

16. COSMOS Results: Benchmark Problems
16
Med vs. Max Med vs. Std-dev Med vs. Med-extra
0 0.05 0.1 0.15 0.2
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
Mode of Median Error, E
mo
med
max
mo
Mode of Maximum Error, E
HP-0
HP-1
HP-2
Pareto
0 0.05 0.1 0.15 0.2
0.2
0.18
0.16
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
Mode of Median Error, E
mo
med
med,
mo
Mode of Median Error at 20% more samples, E
HP-0
HP-1
HP-2
Pareto
1.4
1.2
1
0.8
0.6
0.4
0.2
0
0.02 0.04 0.06 0.08 0.1 0.12
Mode of Median Error, E
mo
med
med

Standard Deviation of Median Error, E
HP-0
HP-1
HP-2
Pareto
Branin Hoo (2D)
2.4
2
1.6
1.2
0.8
0.4
0.2 0.4 0.6 0.8 1 1.2
Mode of Median Error, E
mo
med
max
mo
Mode of Maximum Error, E
HP-0
HP-1
HP-2
Pareto
40
35
30
25
20
15
10
5
0
Hartman-6 (6 D)
0.2 0.4 0.6 0.8 1 1.2
Mode of Median Error, E
mo
med
med

Standard Deviation of Median Error, E
HP-0
HP-1
HP-2
Pareto
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.2 0.4 0.6 0.8 1 1.2
Mode of Median Error, E
mo
med
med,
mo
Mode of Median Error at 20% more samples, E
HP-0
HP-1
HP-2
Pareto
Med vs. Max Med vs. Std-dev Med vs. Med-extra

17. COSMOS Results: Benchmark Problems
mo
0.35 0.4 0.45 0.5
0.11
0.105
0.1
0.095
0.09
0.085
0.08
0.075
0.07
0.065
0.06
0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
mo
Mode of Median Error, E
med
med,
med,
mo
mo
Mode of Median Error at 20% more samples, E
HP-0
HP-1
HP-2
Pareto
mo
0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7
0.35
0.34
0.33
0.32
0.31
0.3
0.29
0.28
0.27
0.26
0.25
3.5
3
2.5
2
1.5
1
0.5
0
mo
Mode of Median Error, E
med
mo
max
max
mo
Mode of Maximum Error, E
HP-0
HP-1
HP-2
Pareto
0.09 0.1 0.11 0.12 0.13 0.14 0.15
Mode of Median Error, E
med
Mode of Median Error at 20% more samples, E
HP-0
HP-1
HP-2
Pareto
Dixon & Price (30D)
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0.09 0.1 0.11 0.12 0.13 0.14 0.15
Mode of Median Error, E
mo
med
med

Standard Deviation of Median Error, E
HP-0
HP-1
HP-2
Pareto
0.09 0.1 0.11 0.12 0.13 0.14 0.15
Mode of Median Error, E
med
Mode of Maximum Error, E
HP-0
HP-1
HP-2
Pareto
17
Med vs. Max Med vs. Std-dev Med vs. Med-extra
Neumaier Perm (20D)
Med vs. Max Med vs. Std-dev Med vs. Med-extra

18. COSMOS Results: Airfoil Problem
18
Med vs. Max Med vs. Std-dev Med vs. Med-extra
0 0.005 0.01 0.015 0.02
0.035
0.03
0.025
0.02
0.015
0.01
0.005
0
Mode of Median Error, E
mo
med
max
mo
Mode of Maximum Error, E
HP-0
HP-1
HP-2
Pareto
0.02
0.018
0.016
0.014
0.012
0.01
0.008
0.006
0.004
0.002
med,
mo
퐶퐿
퐶퐷
= 푓 푥1, 푥2, 푥3, 훼
mo
High-fidelity samples generated
by a Fluent CFD simulation
0 0.005 0.01 0.015 0.02
0.02
0.018
0.016
0.014
0.012
0.01
0.008
0.006
0.004
Mode of Median Error, E
mo
med
med

Standard Deviation of Median Error, E
HP-0
HP-1
HP-2
Pareto
0 0.005 0.01 0.015 0.02 0.025
0
Mode of Median Error, E
med
Mode of Median Error at 20% more samples, E
HP-0
HP-1
HP-2
Pareto

19. COSMOS Results: Summary
 Widely different sets of surrogates models were selected as the
optimum set in the five different problems.
 A diverse set of surrogate-kernel combinations are often observed
to provide important trade-offs.
19
The Best Trade-off Models

20. Concluding Remarks
 A new framework, COSMOS, was developed to select surrogate models
based on criteria driven by user preference (e.g., median or max error).
 COSMOS can identify optimal model-kernel combinations from a large
pool of candidates, by using
1. The model-independent error measures given by PEMF, and
2. A novel MINLP formulation.
 On applying COSMOS to a suite of benchmark test problems, we found:
1. Same surrogate-kernel combinations can yield a noticeable spread of best trade-offs
(at different HP values);
2. Diverse surrogate models often constitute the set of best trade-off models.
 These initial tests readily exhibit the need for such frameworks for
automated selection of globally competitive surrogates.
 Future research directions: Application to more complex practical
problems, and a smarter apriori sorting of the model-kernel candidates.
20

21. Questions
and
Comments
21
Thank you

22. COSMOS: MATLAB-based GUI
22
For those interested to contribute models or test problems,
or interested to try out COSMOS,
please contact chowdhury@bagley.msstate.edu

23. Surrogate-Kernel Combinations
23

24. 24
Median of RAEs
Predictive Estimation of Model Fidelity
(PEMF)
Intermediate Actual model
surrogate model
ε = 풎풆풅 |
풇풊 − 풇풊
풇풊
| ,
푋 = 푋푖푛 + 푋표푢푡
푋푡
퐢 = ퟏ, … , #{푿푻푬}
푇푅 = 푋표푢푡 + {훽푘 }
A chi-square, 흌 ퟐ,goodness-of-fit criterion
푋푡
푇퐸 = X − {푋푡
푇푅 }
kth subset of
inside-region
sample points
Inside and outside sets
Momed
It. 2
It. 1
t1 t2 t3 t4
Number of Training Points
휒 2 =
푚
푖=1
(표푖 − 푡푖 )^2
푡푖
It. 4
Predicted
Median Error
Mean Error
No. of Training points
Mode of Median Error
No. of Training points
Branin-Hoo Function (RBF)
It. 3
퐹 푛푡 = 푎0(푛푡 )−푎1
OR
퐹 푛푡 = 푎0 푒−푎1(푛푡) 
Model Based Systems
Design
Integrative Modeling and
Design of Wind Farms
Energy-Sustainable Smart
Buildings
Reconfigurable Unmanned
Aerial Vehicles (UAV)
We randomly divide the set of sample points into intermediate sets of
1.Training points and
2.Test points

25. Comparison
25

26. Comparing Computational Time
26
One step method requires around 1/7th the time in
searching for optimal models.