SciRepro Workshop, TPDL, 9 Sept 2016, Hannover, Germany

1. What is
Reproducibility?
The R* brouhaha
(and how Research Objects
can help)
Professor Carole Goble
The University of Manchester, UK
Software Sustainability Institute, UK
ELIXIR-UK, FAIRDOMAssociation e.V.
carole.goble@manchester.ac.uk
First International Workshop on Reproducible Open Science @ TPDL, 9 Sept 2016, Hannover, Germany

2. Acknowledgements
• Dagstuhl Seminar 16041 , January 2016
– http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16041
• ATI Symposium Reproducibility, Sustainability and Preservation , April 2016
– https://turing.ac.uk/events/reproducibility-sustainability-and-preservation/
– https://osf.io/bcef5/files/
• CTitus Brown
• Juliana Freire
• David De Roure
• Stian Soiland-Reyes
• Barend Mons
• Tim Clark
• Daniel Garijo
• Norman Morrison

3. “When I use a word," Humpty Dumpty
said in rather a scornful tone, "it means
just what I choose it to mean - neither
more nor less.”
Carroll, Through the Looking Glass
re-compute
replicate
rerun
repeat
re-examine
repurpose
recreate
reuse
restore
reconstruct review
regenerate
revise
recycle
redo
robustness
tolerance
verificationcompliancevalidation assurance
remix

4. Reproducibility of
Reproducibility Research

5. Computational Science
http://tpeterka.github.io/maui-project/
From:The Future of ScientificWorkflows, Report of DOEWorkshop 2015,
http://science.energy.gov/~/media/ascr/pdf/programdocuments/docs/workflows_final_report.pd
1. Observational,
experimental
2. Theoretical
3. Simulation
4. Data intensive

6. BioSTIF
Computational
Science

7. Scientific publications goals:
(i) announce a result
(ii) convince readers its correct.
Papers in experimental science
should describe the results and
provide a clear enough protocol to
allow successful repetition and
extension.
Papers in computational science
should describe the results and
provide the complete software
development environment, data
and set of instructions which
generated the figures.
VirtualWitnessing*
*Leviathan and theAir-Pump: Hobbes, Boyle, and the
Experimental Life (1985) Shapin and Schaffer.
Jill Mesirov
David Donoho

8. Datasets, Data collections
Standard operating procedures
Software, algorithms
Configurations,
Tools and apps, services
Codes, code libraries
Workflows, scripts
System software
Infrastructure
Compilers, hardware
Systems of
Systems
Heterogeneous hybrid
patchwork of tools and
service evolving over time

9. 10 “Simple” Rules for Reproducible
Computational Research: RACE
1. For Every Result, Keep Track of How It Was
Produced
2. Avoid Manual Data Manipulation Steps
3. Archive the Exact Versions of All External
Programs Used
4. Version Control All Custom Scripts
5. Record All Intermediate Results, When Possible in
Standardized Formats
6. For Analyses That Include Randomness, Note
Underlying Random Seeds
7. Always Store Raw Data behind Plots
8. Generate Hierarchical Analysis Output, Allowing
Layers of Increasing Detail to Be Inspected
9. Connect Textual Statements to Underlying
Results
10. Provide Public Access to Scripts, Runs, and
Results
Sandve GK, Nekrutenko A,Taylor J, Hovig E (2013)Ten Simple Rules for Reproducible
Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285
Record
Everything
Automate
Everything
Contain
Everything
Expose
Everything

10. Preparation pain
independent testing trials and tribulations
[Norman Morrison]
replication hostility no funding, time, recognition, place to publish
resource intensive access to the complete environment

11. Lab Analogy: Witnessing “Datascopes”
Input Data
Software
Output Data
Config
Parameters
Methods
techniques, algorithms,
spec. of the steps, models
Materials
datasets, parameters,
algorithm seeds
Instruments
codes, services, scripts,
underlying libraries,
workflows, , ref resources
Laboratory
sw and hw infrastructure,
systems software,
integrative platforms
computational environment

12. “Micro” Reproducibility
“Macro” Reproducibility
Fixivity
Validate
Verify
Trust

13. Repeat, Replicate, Robust
[CTitus Brown]
https://2016-oslo-repeatability.readthedocs.org/en/latest/repeatability-discussion.html
Why the differences?
Reproduce,Trust

14. “an experiment is reproducible until
another laboratory tries to repeat it”
Alexander Kohn
Repeatability:
“Sameness”
Same result
1 Lab
1 experiment
Reproducibility:
“Similarity”
Similar result
> 1 Lab
> 1 experiment
Validate
Verify

15. Method
Reproducibility
the provision of
enough detail about
study procedures and
data so the same
procedures could, in
theory or in actuality,
be exactly repeated.
Result Reproducibility
(aka replicability)
obtaining the same
results from the
conduct of an
independent study
whose procedures are
as closely matched to
the original experiment
as possible
What does research reproducibility mean? Steven N. Goodman, Daniele Fanelli, John
P. A. Ioannidis ScienceTranslational Medicine 8 (341), 341ps12.
[doi: 10.1126/scitranslmed.aaf5027]
http://stm.sciencemag.org/content/scitransmed/8/341/341ps12.full.pdf

16. Productivity
Track differences
Validate
Verify

17. reviewers want additional work
statistician wants more runs
analysis needs to be repeated
post-doc leaves,
student arrives
new/revised datasets
updated/new versions of
algorithms/codes
sample was contaminated
better kit - longer simulations
new partners, new projects
Personal & Lab
Productivity
Public Good
Reproducibility

18. “Datascope” Lab Analogy
Methods
techniques, algorithms,
spec. of the steps, models
Materials
datasets, parameters,
algorithm seeds
Instruments
codes, services, scripts,
underlying libraries,
workflows, ref datasets
Laboratory
sw and hw infrastructure,
systems software,
integrative platforms
computational environment

19. “Datascope” Lab Analogy
Methods
techniques, algorithms,
spec. of the steps, models
Materials
datasets, parameters,
algorithm seeds
Instruments
codes, services, scripts,
underlying libraries,
workflows, ref datasets
Laboratory
sw and hw infrastructure,
systems software,
integrative platforms
computational environment
Form
Function

20. “Datascope” Practicalities
Methods
techniques, algorithms,
spec. of the steps, models
Materials
datasets, parameters,
algorithm seeds
Instruments
codes, services, scripts,
underlying libraries,
workflows, ref datasets
Laboratory
sw and hw infrastructure,
systems software,
integrative platforms
computational environment
Living Dependencies
Science,
methods,
datasets
questions stay,
answers change
breakage, labs
decay, services and
techniques come
and go, new
instruments,
updated datasets,
services, codes,
hardware
One offs, streams,
stochastics,
sensitivities,
scale, non-portable
data
black boxes
supercomputer
access
non-portable
software
licensing restrictions
unreliable resources
black boxes
complexity

21. T1 T2
evolving ref datasets,
new simulation codes
Environment
Archived vs Active
Contained vs Distributed
Regimented vs Free-for-all
Who owns the dependencies?
Dependencies -> Manage
Black boxes -> Expose
Dynamics -> Fixity
Reliability

22. Replicate harder than Reproduce?
Repeating the experiment or the set up?
Container Conundrum Results willVary
ReplicabilityWindow
All experiments become less replicable over time
Prepare to repair

23. Levels of Computational Reproducibility
Coverage: how
much of an
experiment is
reproducible
OriginalExperimentSimilarExperimentDifferentExperiment
Portability
Depth: how much of an experiment is available
Binaries +
Data
Source Code /
Workflow
+ Data
Binaries +
Data +
Dependencies
Source Code /
Workflow
+ Data +
Dependencies
Virtual Machine
Binaries +
Data +
Dependencies
Virtual Machine
Source Code /
Workflow
+ Data +
Dependencies
Figures +
Data
[Freire, 2014]
Minimum:
data and source
code available
under terms
that permit
inspection and
execution.

24. Measuring Information Gain from Reproducibility
Research goal
Method/Alg.
Platform/Exec Env
Data Parameters
Input data
Actors
Information Gain
Implementation/Code
No change
Change
Don’t care
https://linkingresearch.wordpress.com/2016/02/21/dagstuhl-seminar-report-reproducibility-of-data-oriented-experiments-in-e-scienc/
http://www.dagstuhl.de/16041

25. How? Preserve by Reporting, Reproduce by Reading
Archived
Record
Description Zoo
standards, common metadata

26. How? Preserve by Maintaining, Repairing, Containing
Reproduce by Running, Emulating, Reconstructing
Active Instrument Byte level Buildability Zoo

27. provenance
portability, preservation
robustness, versioning
access description
standards
common APIs
licensing, identifiers
standards,
common metadata
change
variation sensitivity
discrepancy handling
packaging, containers
FAIR RACE Reproducibility Dimensions
dependencies
steps

28. Research Object
Standards-based metadata framework for logically and
physically bundling resources with context,
http://researchobject.org
Bigger on the inside than the outside
external referencing

29. Manifest
Construction
Aggregates
link things together
Annotations
about things & their
relationships
Container
Research Object Standards-based metadata framework for logically
and physically bundling resources with context, http://researchobject.org
Packaging content & links:
Zip files, BagIt, Docker images
Catalogues & Commons Platforms:
FAIRDOM
Manifest
Description
Dependencies
what else is
needed
Versioning
its evolution
Checklists
what should
be there
Provenance
where it
came from
Identification
locate things
regardless where
id

30. Systems Biology
Commons
• Link data, models
and SOPs
• Standards
• Span resources
• Snapshot + DOIs
• Bundle and export
• Logical bundles

31. Belhajjame et al (2015) Using a suite of ontologies for preserving workflow-centric research objects,
JWeb Semantics doi:10.1016/j.websem.2015.01.003
application/vnd.wf4ever.robundle+zip
Workflow Research Objects
exchange, portability and
maintenance
*https://2016-oslo-repeatability.readthedocs.org/en/latest/overview-and-agenda.html

32. Asthma Research e-Lab
Dataset building and
releasing
Standardised
packing of Systems
Biology models
European Space
Agency RO Library
Large dataset
management for life
science workflows
LHC ATLAS
experiments
Notre Dame U Rostock
Encyclopedia of DNA
Elements
PeptideAtlas

33. Words matter.
Reproducibility is not a end.
Its a means to an end.
Beware reproducibility zealots.
50 Shades of Reproducibility.
form vs function
A conundrum:
big co-operative data-driven
science makes reproducibility
desirable but also means
dependency and change are to be
expected.
Lab analogy for
computational science

34. Bonus Slides