Classifying and visualizing the disciplinary focus of universities: The invisible factor of university rankings
•
2 likes•12,297 views
Presentation of the PhD Defense of Nicolas Robinson-Garcia which took place in the Faculty of Information Science and Communication in the University of Granada (Spain) on July 14, 2014.
Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Classifying and visualizing the disciplinary focus of universities: The invisible factor of university rankings
Similar to Classifying and visualizing the disciplinary focus of universities: The invisible factor of university rankings (20)
More from Nicolas Robinson-Garcia
More from Nicolas Robinson-Garcia (20)
Recently uploaded
Recently uploaded (20)
Classifying and visualizing the disciplinary focus of universities: The invisible factor of university rankings
- 1. Classifying and visualizing the disciplinary focus of universities THE INVISIBLE FACTOR OF UNIVERSITY RANKINGS PhD Candidate: Nicolás Robinson-García Thesis Defense July 14, 2014 Supervisors: Evaristo Jiménez-Contreras & Emilio Delgado López-Cózar University of Granada
- 2. Rationale Background Research questions Contributions Conclusions Personal motivation The I-UGR Rankings aim to complement international rankings and account for disciplinary differences. My contribution to this project is reflected in this thesis. 2 1th edition published in 2010
- 3. Rationale Background Research questions Contributions Conclusions Personal motivation 3
- 4. Rationale Background Research questions Contributions Conclusions Rankings as bibliometric tools 4 […] rankings tend to emphasize vertical differences between institutions […]. At the same time, they obscure horizontal differences. Marginson, 2007 […] ‘mediocrity’ of a university does not necessarily imply that it is mediocre in all disciplines. Moed, 2008
- 5. Rationale Background Research questions Contributions Conclusions Rankings in research policy 5 International rankings are focused on World-Class universities However, many research managers use them to evaluate their national system
- 6. Rationale Background Research questions Contributions Conclusions Table of contents 6 Background • Bibliometrics for evaluating • National Performance research funding systems • International Rankings Research questions • Are national university rankings necessary in an international context? • How can we develop tools to identify disciplinary profiles? Contributions • Complementing international rankings • Mapping techniques for interpreting rankings Conclusions • Discussion • Implications
- 7. 7 Rationale Background Research questions Contributions Conclusions Bibliometrics as an evaluation tool The need for bibliometric indicators arises from the enormous growth in scientific and technological activity that has been occuring for centuries, and has now reached avalanche proportions. Narin et al., 1994 Type of indicator Type of measurement Unit of analysis Papers Citations Usage Data Altmetrics … Activity measurement Impact measurement Linkage measurement Micro level Meso level Macro level
- 8. 8 Rationale Background Research questions Contributions Conclusions Why the need to assess universities’ performance I. Research as the Second Mission of universities II. The expansion of Big Science III. Knowledge-based economies and the production of human capital in a globalized context IV.Economic constraints and the need for accountability
- 9. 9 Rationale Background Research questions Contributions Conclusions National funding systems and International Rankings
- 10. 10 Rationale Background Research questions Contributions Conclusions The problem with international rankings
- 11. 11 Rationale Background Research questions Contributions Conclusions The problem with international rankings Rankings consider universities as homogeneous, neglecting their disciplinary differences Health & Life Sciences Natural & Exact Sciences Social SciencesHumanities
- 12. 12 Rationale Background Research questions Contributions Conclusions The problem with international rankings Can we use international rankings to analyze national systems?
- 13. 13 Rationale Background Research questions Contributions Conclusions The problem with international rankings Thus, the country in which a university is located plays an important role in the performance of a university Bornmann, Mutz & Daniel, 2013 Countries typically have legal and institutional rules within which their universities function Aghion et al., 2010
- 14. Rationale Background Research questions Contributions Conclusions Table of contents 14 Background • Bibliometrics for evaluating • National Performance research funding systems • International Rankings Research questions • Are national university rankings necessary in an international context? • How can we develop tools to identify disciplinary profiles? Contributions • Complementing international rankings • Mapping techniques for interpreting rankings Conclusions • Discussion • Implications
- 15. Rationale Background Research questions Contributions Conclusions 15 RQ 1.Are national university rankings necessary in an international context? • An insight into the importante of national rankings in an international context • What do university rankings by fields rank?
- 16. Rationale Background Research questions Contributions Conclusions 16 RQ 2. How can we develop tools for research policy-makers that can allow them to identify disciplinary profiles? • What do university rankings by fields rank? • On the use of biplot analysis • Mapping academic institutions according to their journal publication profile • Análisis de redes de las universidades españolas
- 17. Rationale Background Research questions Contributions Conclusions Table of contents 17 Background • Bibliometrics for evaluating • National Performance research funding systems • International Rankings Research questions • Are national university rankings necessary in an international context? • How can we develop tools to identify disciplinary profiles? Contributions • Complementing international rankings • Mapping techniques for interpreting rankings Conclusions • Discussion • Implications
- 18. 18 Rationale Background Research questions Contributions Conclusions An insight into the importance of national university rankings in an international context: The case of the I-UGR Rankings of Spanish Universities Research questions > Are national university rankings necessary, considering the existence of international rankings? > Levels of agreement between national and international rankings regarding their coherence and concordance Methodology > Rankings analyzed: Shanghai Ranking – QS Ranking – NTU Ranking – Leiden Ranking – I-UGR Rankings > Indicators: Spearman rank correlation – Coincidence of top Spanish universities
- 19. 19 Rationale Background Research questions Contributions Conclusions An insight into the importance of national university rankings in an international context: The case of the I-UGR Rankings of Spanish Universities Conclusions > Spain is poorly represented in international rankings 25% of the Spanish system > Despite methodological differences, national rankings are a necessary complement of international university rankings >We must consider these methodological differences derived from the construction of fields as well as the indicator used
- 20. 20 Rationale Background Research questions Contributions Conclusions What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications Research questions > Do rankings by fields represent the structure of universities? > Can we provide indicators that show the degree of correspondence between each field and organizational unit? Methodology > Universities analyzed: Pompeu Fabra – Granada > Data processing: Identification of organizational units > Indicators: Betweenness Centrality – Gini Index – Number of subject – Number of disciplines
- 21. 21 Rationale Background Research questions Contributions Conclusions What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications Results Faculties Departments University of Granada
- 22. 22 Rationale Background Research questions Contributions Conclusions What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications Results Faculties Departments Pompeu Fabra University
- 23. 23 Rationale Background Research questions Contributions Conclusions What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications Conclusions >Working with addresses adopting a top down approach has implicit problems of difficult solution >The structure and size of universities varies significantly from one to other which prevent from using organizational structures to construct fields >The Gini coefficient and the Betweenness Centrality are useful indicators for pointing out discrepancies between fields and organizational units
- 24. 24 Rationale Background Research questions Contributions Conclusions How can we identify disciplinary profiles of universities? Using mapping techniques to analyze the multidimensional nature of universities
- 25. 25 Rationale Background Research questions Contributions Conclusions On the use of biplot analysis for multivariate bibliometric and scientific indicators Objective > Present the visualization methodology known as Biplot Analysis Methodology >Three case studies: European countries – Universities – Scientific fields >Validation: Comparison with other visualization techniques: CA – MDS – PCA
- 26. 26 Rationale Background Research questions Contributions Conclusions On the use of biplot analysis for multivariate bibliometric and scientific indicators
- 27. 27 Rationale Background Research questions Contributions Conclusions On the use of biplot analysis for multivariate bibliometric and scientific indicators Comparison with other techniques
- 28. 28 Rationale Background Research questions Contributions Conclusions On the use of biplot analysis for multivariate bibliometric and scientific indicators Case study 2. Universities
- 29. 29 Rationale Background Research questions Contributions Conclusions On the use of biplot analysis for multivariate bibliometric and scientific indicators Conclusions > Biplot analysis may well be an important analysis tool for bibliometric studies >They are easy to read tools of potential use in the research policy arena > As well as representing the relations between cases, they also represent relations between variables, and variables and cases
- 30. 30 Rationale Background Research questions Contributions Conclusions Mapping academic institutions according to their journal publication profile: Spanish universities as a case study Objective > Present a novel methodology for representing universities according to their journal publication profile Structure > Methodological proposal > Case study with Spanish universities
- 31. 31 Rationale Background Research questions Contributions Conclusions Mapping academic institutions according to their journal publication profile: Spanish universities as a case study Hypothesis Universities with a similar research profile should publish in the same scientific journals
- 32. 32 Rationale Background Research questions Contributions Conclusions Mapping academic institutions according to their journal publication profile: Spanish universities as a case study Information and Communication Technologies Collaboration and geographical proximity Basic vs.Applied 40-50% Q1 journals 30-40% Q1 journals <30% Q1 journals
- 33. 33 Rationale Background Research questions Contributions Conclusions Mapping academic institutions according to their journal publication profile: Spanish universities as a case study Conclusions > Mapping techniques end with limitations derived from a rigid classification system of universities > Science maps combined with impact or visibility indicators shows vertical stratification and horizontal diversification >The technique highlights geographical, social and/or historical relationships > Limitation: Too many links between universities blur similarities between low perfomance universities
- 34. 34 Rationale Background Research questions Contributions Conclusions Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas Objectives > Map and analyze the research perfomance of Spanish universities according to their journal publication profile by areas > Analyze the role of universities according to their centrality values Methodology > Unit of analysis: Spanish universities with at least 125 publication in the 2007-2011 time period > Indicators: Production – % Q1 publications – Activity Index – Closeness centrality > Areas: Social Sciences – Exact Sciences – Engineering & Technology – Life Sciences – Health Sciences
- 35. 35 Rationale Background Research questions Contributions Conclusions Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas Economics & Business Social Sciences >30% Q1 journals 15-30% Q1 journals <15% Q1 journals
- 36. 36 Rationale Background Research questions Contributions Conclusions Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas HighTransitivity!! Exact Sciences >55% Q1 journals 40-55% Q1 journals <40% Q1 journals
- 37. 37 Rationale Background Research questions Contributions Conclusions Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas Engineering & Technology >60% Q1 journals 35-60% Q1 journals <35% Q1 journals Two communities Geographical proximity
- 38. 38 Rationale Background Research questions Contributions Conclusions Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas Life Sciences >60% Q1 journals 45-60% Q1 journals <45% Q1 journals Core universities
- 39. 39 Rationale Background Research questions Contributions Conclusions Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas Health Sciences >45% Q1 journals 25-45% Q1 journals <25% Q1 journals Cluster 1 Cluster 2 Cluster 3
- 40. 40 Rationale Background Research questions Contributions Conclusions Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas Characteristics Examples Class I •Large size •Generalist •Higher centrality values Barcelona Complutense Autónoma de Madrid… Class 2 •High specialization in a certain area or research field Politécnica de Cataluña Navarra Carlos III Madrid… Class 3 •High specialization in a certain area •Different publication profile (outlier) Pompeu Fabra Class 4 •Small size •Poor research performance Burgos León Jaén… Spanish universities according to their disciplinary focus
- 41. 41 Rationale Background Research questions Contributions Conclusions Análisis de redes de las universidades españolas de acuerdo a su perfil de publicación en revistas por áreas científicas Conclusions > Science maps are powerful tools for research policy makers that complement the information provided by rankings > Four classes of Spanish universities were found by analyzing their journal publication profile > Catalan universities and universities from Madrid prevail in all areas > Pompeu Fabra as an outlier in the Spanish university system
- 42. Rationale Background Research questions Contributions Conclusions Table of contents 42 Background • Bibliometrics for evaluating • National Performance research funding systems • International Rankings Research questions • Are national university rankings necessary in an international context? • How can we develop tools to identify disciplinary profiles? Contributions • Complementing international rankings • Mapping techniques for interpreting rankings Conclusions • Discussion • Implications
- 43. 43 Rationale Background Research questions Contributions Conclusions Rankings and research policy > University rankings respond to a demand for accountable and easy-to-use tools by research managers > If one is to use rankings’ information to analyze research performance they should not focus just on one
- 44. 44 Rationale Background Research questions Contributions Conclusions Are national university rankings necessary in an international context? > International rankings should not be used to assess national university systems as they focus on a non- representative sample of universities > National rankings may complement this information by positioning the rest of the universities in the system > Comparisons should be made by research field in order to draw reliable conclusions
- 45. 45 Rationale Background Research questions Contributions Conclusions The invisible factor > There are interpretation issues on the use of rankings by fields of difficult solution, if any > Science maps may be helpful complementing the information of rankings
- 46. 46 Rationale Background Research questions Contributions Conclusions How can we develop tools to identify disciplinary profiles? > Biplot analysis I. Relate variables and cases II. Analyze multivariate data > Journal publication profile I. Broad definition of the reasons behind the similarity of two institutions II. Not dependent on a classification system
- 47. 47 Rationale Background Research questions Contributions Conclusions What kind of relations are defined by the journal publication profile? > Geographical and historical ties > Disciplinary relations > Research orientation > Publication patterns and outliers > Size dependence ¿?
- 48. 48 Rationale Background Research questions Contributions Conclusions The analysis of disciplinary differences at the institutional level is far too complex for us to be able to reduce it to league tables Science mapping techniques are a good way to complement the information provided by rankings as they are easy-to-read tools [..] they provide far richer information
- 49. 49 ‘Nothing is ever quite true’, said Lord Henry OscarWilde The Picture of Dorian Gray
- 50. Classifying and visualizing the disciplinary focus of universities THE INVISIBLE FACTOR OF UNIVERSITY RANKINGS PhD Candidate: Nicolás Robinson-García Thesis Defense July 14, 2014 Supervisors: Evaristo Jiménez-Contreras & Emilio Delgado López-Cózar
Editor's Notes
- Thank you professor Docampo. Before presenting the thesis itself, I would like to address the motivations that led me to analyze university rankings in general, and especifically the problems derived from their use for research policy issues and their neglecting of disciplinary profiles. I believe that these reasons offer also a good introduction to the thesis.
- This thesis originates from my involvement in the project I-UGR Rankings. The first edition of the I-UGR Rankings took place in 2010 and since then has been updated on annual basis. Among other, they characterize by having a national coverage and, instead of offering a global table league, including 12 rankings by 12 fields and 37 by scientific disciplines.
- The idea behind such presentation, is to avoid the reader to even consider universities as a whole on the one hand, and on the other, offer an analysis of Spanish university system. Although there are other aspects worth mentioning, these are the two which I will address here.
- The first limitation or aspect relates to the incapability rankings have to consider institutional diversity. These tools seem to be quite straightforward on their interpretation. A university is either better or worse than an other. However, this type of analysis only makes sense the units of analysis are homogeneous between them, which is not the case. In this sense, the I-UGR Rankings propose the introduction of rankings by fields or disciplines, instead than global league tables as a means to reflect disciplinary differences between universities. So that seems as a way of solving this issue.
- The other aspect I analyze has to do with the danger of using international rankings, which are focused in most cases on top universities, to evaluate national university systems. Especially in countries where universities do not comply the necessary requirements as to allow them to compete in an international context. Nowadays we are more than accostumed in Spain to read how the lack of universities in these rankings is an evidence of the defficiencies of the system. Inferring any type of decision based on tools which were not designed for comparing countries entails many dangers in the higher education scenario.
- So, I have given you the context, now I’ll proceed with the thesis itself. The structure I will follow is very similar to that of the thesis itself. First I will give a brief overview on the expansion of bibliometrics and their role in research evaluation and the development of international rankings. Second, I wil pose the research questions which go in accord with the two aspects presented before. I will then briefly comment each of the papers presented. I will conclude wrapping up how these papers respond to the aims presented and the implications they may or could have.
- The field of bibliometrics encompasses many other aspects, however and without any question it is its application for research evaluation the one that has had a greater impact. Although the term evaluative bibliometrics was coined by Narin in the late 1970s, bibliometric indicators play nowadays a key role on research funding allocation in many countries. Of course, there use is surrounded by much controversies, but despite the many types indicators and alternatives and complements, citations and publications remain the main type indicators used for research evaluation. Rankings implement activity measurement and impact measurement focusing at the institutional level.
- At this point, one may question why should we evaluate universities’ research performance. In summary, the relevance research currently has is a consequence of the investment needed to support it, derived from the changes already highlighted by De Solla Price in his book Little Science, Big Science and beyond, as well as the changes derived from globalization and knowledge-based economies. Also, the different economic crises that have taken place in the 1980s and now empasized by the current crisis, urge research policy makers to find supposedly ‘objective’, ‘transparent’ and ‘accountable’ measures on which to base decisions.
- Here I show a timeline of different milestones related with the emergence of national funding systems and university rankings. It goes back to the beginning of the 20th Century and the first precedents to university rankings, And it ends at the beginning of the 21st Century. Here you see how there is an interest in the 1960s to evaluate research and doctoral programs in the USA and national university rankings start to emerge. In the 1980s the first national funding systems appear in the UK and later on Spain, expanding to other countries like Japan, Hong Kong but also Australia which does appear in the slide, Italy, etc. Until the end of the 20th Century evaluation exercises have a national scope, and with some exceptions, they are directed at universities. The main change takes place in 2003 when the Shanghai Rankings kicks off its first edition and suddenly universities and university systems are confronted for the first to competition at an international level. Since then, of course many other similar rankings have emerged.
- Although many of them claimed to analyze the teaching and research quality of universities. The truth is that they all rely heavily on bibliometric indicators. Some of them exclusively. We must point out the role played by Scopus and Web of Science who are the main suppliers of the bibliometric data. The first rankings (Shanghai, Times or QS), took by surprise to the bibliometric community. The authors were completely unknown and made a sloppy use of bibliometric indicators, which were severely criticized as they seemed to ignore some of the important methodological limitations of research analysis at the institutional level.
- One of them is their neglecting of the disciplinary focus of universities. While rankings establish vertical comparisons, they not only ignore horizontal differences but also difference on the nature of research.
- Despite criticisms, research managers rapidly included rankings in their toolbox and some studies emerged extrapolating rankings’ results to analyze national university systems (clic). This meant, not only considering universities as homogeneous, but also considering that they all abide by the same set of rules and pursue the same objectives.
- Hence, and this would be a first conclusion after analyzing the scientific literature: it is important to consider that rankings are not designed to such purposes, and even more, the purposes pursued by these tools is not always clear. Therefore, we must understand not only how rankings function, but how the system or universities function, and what is it we want to achieve when we analyze them.
- Now I will give an overview of the five contributions presented and their main findings.
- Therefore the research questions I intend to analyze and solve are two. The first one, is directed as these latter aspect considered, that is, how can we analyze national university systems and position their performance in comparison with the rest of the world. The first contribution: “An insight into the importance of national university ranking in an international context” presents an analysis in this line of thought. The second contribution, “What do university rankings by fields rank? Discrepancies between the organizational structure of universities and bibliometric classifications” also explores pecularities at the institutional level.
- In fact, it presents the reasons why science mapping techniques could be useful for complementing the information provided by rankings, and hence, respond to the second research question ‘How can we develop tools for research policy-makers that can allow them to identify disciplinary profiles?’. Here we present the biplot analysis visualization technique for multivariate data and the journal publication profile methodology. It will be this last methodology the one employed in the last contribution, ‘Network analysis of Spanish universities according to their journal publication profile by scientific areas’. Here we will propose a classification of Spanish universities according to their research performance.
- Now I will give an overview of the five contributions presented and their main findings.
- So how can we solve these interpretation issues and develop tools that allow research policy makers to identify disciplinary profiles