A paper presented at ESSA 2013, Warsaw. Abstract: We consider the problem of finding suitable measures to validate simulated networks as outcome of (agent-based) social simulations. A number of techniques from computer science and social sciences are reviewed in this paper, which tries to compare and ‘fit’ various simulated networks to the avail-able data by using network measures. We look at several social network analy-sis measures but then turn our focus to techniques that not only consider the po-sition of the nodes but also their characteristics and their tendency to cluster with other nodes in the network – subgroup identification. We discuss how stat-ic and dynamic nature of networks may be compared. We conclude by urging a more comprehensive, transparent and rigorous approach to comparing simula-tion-generated networks against the available data.

1. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 1
Towards Validating Social Network
Simulations
SMA Abbas1, Shah Jamal Alam2 and Bruce Edmonds1
1Centre for Policy Modelling, Manchester Metropolitan University
2School of Geosciences, University of Edinburgh

2. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 2
The Situation
A Simulation
Social
“System”
Generates
Measured
Are these
“essentially” the
same?
A Class of Networks Another Class of Networks
?

3. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 3
Key questions
• What properties of the synthetic networks, one
would expect to observe given how the model has
been constructed
• Which of these properties are „significant‟ in terms
of the intended processes in the model
• Which class of target networks one might expect
to observe if one could “re-run” reality under the
same basic conditions as assumed in the model
• Do these classes match in important respects
• How do we know they do given we only have
samples of synthetic and target networks

4. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 4
The Problem

5. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 5
Summary of Issue
• The space of possible networks is vast
• But many networks will look similar to us, because
our brains can not deal with them but
automatically simplifies them as part of perception
• We are not dealing with single networks but
classes of networks…
• …though these classes are often implicit when a
single network stands for that class (somehow)
• However, in principle, if synthetic and target
networks do match (in some way) then this is
potentially a strong validation

6. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 6
A Cautionary Tale – comparing two network
models
Papadopoulos et al. (2012) Popularity versus similarity in
growing networks. Nature, 489:537-540.

7. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 7
But when compared in a different way…

8. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 8
Some Network Comparison Approaches
Different kinds of things to compare:
• Network Measures
• Network Distributions
• Eigenvalue/Eigenvectors
• Subgroup Identification
• Functional Comparison
• Likelihood of being described by an Exponential
Random Graph Model
• Motif Prevalence

9. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 9
Examples that follow are from work of S.M.A. Abbas
(see papers at https://sites.google.com/site/maliabbas)
An Example of Validating Synthetic vs. Target Networks

10. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 10
An example of comparing measures

11. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 11
An example of comparing distributions
1 5 10 50 100 500 1000
5e-045e-035e-025e-01
Log-log Plot of Degree Distribution
Degree
CumulativeFrequency
Reference
Random
FAOF
Party
Hybrid

12. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 12
Silo Index Comparison
Correlation
0.83
-1.0 -0.9 -0.8 -0.7 -0.6 -0.5
-1.00-0.85-0.70
Dorm Silo Index
Reference Dorm
HybridDorm
Correlation
0.93
-1.00 -0.90 -0.80 -0.70
-1.00-0.90-0.80
Major Silo Index
Reference Major
HybridMajorCorrelation
0.84
-1.0 -0.5 0.0 0.5
-1.0-0.8-0.6
Year Silo Index
Reference Year
HybridYear
Correlation
0.29
-1.0 -0.8 -0.6
-1.00-0.90
High School Silo Index
Reference High School
HybridHighSchool
Reference vs. Hybrid Mode Silo Indices

13. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 13
Assortativity Mixing

14. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 14
Problems in the Literature
Authors are often not clear about:
• Precisely what the links in a synthetic network are
supposed to represent (in terms that would allow an in
principle measurement of observed actors)
• Which aspects of the target network are subject to
measurement error (or otherwise judged not to be
significant) and which should be reproduced by a
synthetic network
• Which aspects of the synthetic network are significant
in terms of the generating process (and which are
essentially accidental)
Readers often cannot judge the extent or meaning of
the match/mismatch between synthetic and target
networks

15. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 15
Conclusions
• Many social simulation models assume stereotypic
networks (e.g. Watts-Strogatz)
• It is increasingly clear that the exact network structure
matters (e.g. Holzhauer ESSA 2013)
• No single „Golden Bullet‟ technique
• More thought needed about what is significant about
the synthetic and target class networks
• Multiple approaches needed to show that classes of
networks are similar – a few 1D measures is not
enough to show this
• Validating networks could be quite a strong validation
of our models…
• …but much more work is needed in this area!

16. Towards Validating Social Network Simulations, SMA Abbas, Shah Jamal Alam, and Bruce Edmonds, ESSA 2013, Warsaw. slide 16
Thanks!
SMA Abbas
https://sites.google.com/site/maliabbas
Shah Jamal Alam
https://sites.google.com/site/jamialam
Bruce Edmonds
http://bruce.edmonds.name
Slides at:
http://www.slideshare.net/BruceEdmonds