Presentation for PDMU-2015

1. DECISION-MAKING IN EDUCATION BASED
ON MULTI-CRITERIA RANKING OF
ALTERNATIVES
THE EFFECT OF SOME UNCERTAINTY
TYPES ON COMPETITIVE SCORES OF HEI
APPLICANTS
Bakhrushin V.E., Ignakhina M.A.
Classic Private University, Zaporizhia
Vladimir.Bakhrushin@gmail.com

2. Methods of multi-criteria ranking
Methods of multi-criteria ranking of alternatives
are widely used in decision making in various areas.
They are based on the traditional approaches of
multi-criteria decision-making. In particular there
are the methods of criteria convolution, the main
criterion, maximin and maximax assessment of
alternatives and others.
But in the case of multi-criteria ranking it is need
to choose not one, but several alternatives that are
the best from a certain point of view.

3. Some examples of linear convolution
Assessments based on the linear convolution of
criteria are the most popular. They are built as
weighted sums of partial indicators.
Well-known examples of such estimates are the
university rankings, competitive scores of Ukrainian HEI
applicants and so on.
It may be assumed that in the nearest future similar
assessment methods will be used for the formation of
state order for specialists training at HEI, allocation of
the scholarships to students, grantmaking for scientific
research etc.

4. Some problems
One of the main problems of multi-criteria
ranking is to ensure validity, reliability and accuracy
of methods used.
These characteristics depend greatly on the
choice of particular criteria, accuracy of
measurements, weighting coefficients, random
fluctuations and other factors.
To reduce the risk of errors some general
principles (for example, the Berlin Principles on
Ranking of HEI) and standardized methods are
used.

5. Some examples of multi-criteria
ranking

6. Some examples of multi-criteria
ranking

7. National Universities Rankings (US
News &World Report)

8. National Universities Rankings (US
News &World Report)

9. National Universities Rankings (US
News &World Report)

10. Forbes Ranking
Index % of total
Student Satisfaction Student evaluations from
RateMyProfessor
Students transfer
25
Post-Graduate Success Salary of alumni; Forbes lists
Nobel and Pulitzer winners,
Guggenheim and MacArthur
Fellows, National Academy of
Sciences members; winners of an
Academy, Emmy, Tony or Grammy
32,5
Student Debts Student' loan debt 25
Graduation Rate Students actually finished their
degrees in four years
7,5
Academic Success Prestigious scholarships and
fellowships
Going on studying for Ph.D.
10

11. Positions of some HEI in ratings
QS ARWU THE Forbs
Amherst College − − − 10
United States Military Academy − − − 9
Harvard University 2 1 2 7
Yale University 4 9 8 6
California Institute of Technology 6 5 1 21
University of California, Berkeley 14 3 6 37
University of California, Los Angeles 17 10 9 44
Columbia University 9 7 10 20
Massachusetts Institute of Technology 1 4 4 5
Pomona College − − − 8
Princeton University 7 6 5 4
Stanford University 3 2 3 2
Swarthmore College − − − 3
Williams College − − − 1
University of Chicago 5 8 7 24

12. Competitive score
Competitive scores of the Ukrainian HEI
applicants are calculated on the basis of external
independent evaluation (EIE) scores. These scores
are actually the quantiles of the normalized
distribution of raw EIE test scores. So they
essentially depend on random variations in the
complexity of test items and the tested applicants’
level of training. In this regard, it is necessary to
analyze the possibility of using the test results of
different years at entrance campaign.

13. Algorithm
initial data (raw test scores) –generated or real;
determination of success thresholds;
excluding the results, that are lower than the
threshold, from the obtained vectors;
calculation of corresponding values of
empirical distribution function as the ratio of its
rank to the vector length;
calculating of outcome scores as the values of
the inverse standard normal distribution function;
transforming of results into 100 – 200 scale.

14. Competitive score

15. Uncertainty factors
unknown complexity of the test, which
changes from year to year;
unknown level of training of graduates, which
also varies from year to year;
unknown not normal distribution of test
results;
unknown relation between the total score and
objective assessment in traditional scale;
…

16. Mathematics / Physics
Year
2007 2008 2009 2010 2010 2011 2012 2012 2013 2013 2014
Mean 28,9 20,1 25,7 36,2 34,7 30,7 37,9 40,1 33,6 29,5 28,6
σ 20,9 14,4 20,5 21,2 21,9 19,3 20,7 20,3 18,6 19,4 20,5
Kα 0,92 0,80 0,90 0,90 0,92 0,89 0,88 0,89 0,88 0,89 0,91
Year
2007 2008 2009 2010 2011 2012 2013 2014
Mean 23,5 35,9 26,5 32,8 29,8 32,0 31,2 27,5
σ 16,1 17,2 14,6 18,7 13,4 16,0 15,6 16,4
Kα 0,82 0,81 0,80 0,86 0,80 0,83 0,81 0,86

17. Mathematics (2011 – 2014)

18. English (2011 – 2014)

19. Model: 1 or 2 thresholds

20. General results
Obtained results show that considered uncertainty
factors may lead to variations of outcome EIE scores in
the range of 2 – 10 points. Ceteris paribus their effect is
more significant in the case of asymmetric and/or non-
uniform distribution of raw scores. The largest
differences are observed at the edges of the distribution
– for the worst and best applicants.
When evaluating possible differences between the
scores of 2015 and previous years, we can expect an
additional variation due to the systematic but not
controlled changes in samples of applicants. They are
associated with a change in the entrance conditions in
2015.