This document summarizes the research of Ludo Waltman on the field of research on research. It discusses algorithms and tools developed by Waltman's group like the Louvain and Leiden algorithms for community detection in networks. It also summarizes Waltman's work analyzing the landscape of science through Dimensions data and identifying the subset of publications focused on research on research. Finally, it shows term maps and analyses of research on research literature in areas like scientometrics, science and technology studies, and innovation studies.

1. The landscape of research on research
Ludo Waltman
Centre for Science and Technology Studies (CWTS), Leiden University
Innovation studies seminar
Utrecht University
October 15, 2019

2. Centre for Science and Technology Studies (CWTS)
• Research center at Leiden University
• Science and technology studies, with an emphasis on scientometrics
• Commercial products and services through CWTS BV
• Our mission: Making the science system better!
1

3. Overview
• Algorithms and tools
• Research on research
• Bridging the gap between theory and data
2

4. Algorithms and tools
3

5. Community detection in networks
4

6. From Louvain to Leiden
5

7. Properties of the Louvain and Leiden algorithms
6
Louvain Leiden
Every iteration
𝛾-separation + +
𝛾-connectivity - +
Stable iteration
Node optimality + +
Subpartition 𝛾-density - +
Asymptotic Subset optimality - +
Subset optimality: There is no subset of nodes in any community
that can be moved to a different (or a new) community

8. Comparison of the Louvain and Leiden algorithms
7

9. VOSviewer
8

10. Research on research
9

11. Research on Research Institute (RoRI)
10

12. Dimensions
11

13. Research funding landscapes
12

14. Creating a landscape of science
• Based on Dimensions data
• 38.8 million publications from the period 1999–2018
• 1.0 billion citation links
• Use of Leiden algorithm for clustering publications based on citation links
• 2890 clusters, each consisting of at least 250 publications
• Use of VOSviewer for visualizing the network of citation links between
clusters
13

15. Landscape of science
14
Social sciences
and humanities
Biomedical and
health sciences
Mathematics and
computer sciencePhysical
sciences
Life and earth
sciences

16. Identifying the research on research literature
• Initial set of 345 manually selected research on research publications
• Expansion based on citation links
• Final set of 2129 research on research publications
15

17. Research on research in the landscape of science
16
Social sciences
and humanities
Biomedical and
health sciences
Mathematics and
computer sciencePhysical
sciences
Life and earth
sciences
scientometrics
science and
technology
studies
innovation
studies
science of
science
higher
education
data sharing
metaresearch
(research quality)
metaresearch
(team science and
research funding)
health
policy
metaresearch
(reproducibility)
metaresearch
(reproducibility)

18. Highly cited publications in scientometrics
• An index to quantify an individual's scientific research output (PNAS, 2005)
• The history and meaning of the journal impact factor (JAMA, 2006)
• The increasing dominance of teams in production of knowledge (Science, 2007)
• Software survey: VOSviewer, a computer program for bibliometric mapping (Scientometrics, 2010)
• Science faculty's subtle gender biases favor male students (PNAS, 2012)
17

19. Highly cited publications in innovation studies
• The dynamics of innovation: from national systems and mode 2 to a triple helix of university-industry-
government relations (Research Policy, 2000)
• Proximity and innovation: A critical assessment (Regional Studies, 2005)
• Knowledge networks as channels and conduits: The effects of spillovers in the Boston biotechnology
community (Organization Science, 2004)
• Links and impacts: The influence of public research on industrial R&D (Management Science, 2002)
• Assessing the impact of organizational practices on the relative productivity of university technology
transfer offices: An exploratory study (Research Policy, 2003)
18

20. Term map of research on research
19

21. Term map of research on research: Scientometrics
20

22. Term map of research on research:
Science and technology studies
21

23. Term map of research on research:
Innovation studies
22

24. Term map of research on research: Metaresearch 1
23

25. Term map of research on research: Metaresearch 2
24

26. Bridging the gap
between theory and
data
25

27. Characterizing scientific fields
26
Humanities
Astronomy and
particle physics
Numberofauthors
Medical
sciences
Reference age
Computer science,
engineering, and
basic life sciences
Social and behavioral
sciences
(mutualdependence?)
(task uncertainty?)

28. Verbs in citing sentences
Knowledge uncertainty
• suggest
• indicate
• show
• demonstrate
• establish
• prove
Strategic uncertainty
• advocate
• argue
• assume
• believe
• claim
• conclude
• hypothesize
27

29. Use of the verb ‘suggest’
28

30. Use of the verb ‘demonstrate’
29

31. Thank you for your attention!
30