ACM promotes reproducible research. In the SIGMM chapter too! This slideshow presents the efforts toward reproducibility at the Multimedia System (MMSys) conference. In particular the action at MMSys'17 of giving reproducible badges to the papers that have made explicit efforts for sharing artifacts (dataset and code source).

1. Reproducibility in ACM
Multimedia Systems
(MMSys) conferences
Gwendal SIMON
(+ many SIGMM conference organizers)

2. 20-23 June 2017 - Taipei (Taiwan)
Sponsor: SIGMM
Support: SIGCOMM, SIGMOBILE, and SIGOPS

3. Open Source and Dataset Track
Submission Reviewing Accept
Research Track
Accepted
papers
Volunteer Validate

4. Open Source and Dataset Track
10/14 accepted papers
All dataset papers (only 3 include code)
Library configuration issues
1

5. Review Form
Impact
Are the researchers in the community likely to use this artifact in their studies? Does
it deal with a topic that currently matters? Does it enable a variety of studies?
Documentation
Do the paper and the associated website provide enough information on how to
use/install/evaluate the artifact? For code source, is the installation procedure
enough to install and use? For dataset, is the structure clearly described?
Validity
Please try to install the artifact and/or to download it and check it. Any unexpected
artifact behavior (depends on the type of artifact such as unexpected output,
scalability issues, crashes, performance variation, bugs, missing information, etc)?

6. Research Track
16/03
Paper
acceptance
notification
19/03
Invitation
to share
artifact
+3 pages
appendix
25/03
Deadline
agreement
to share
artifacts
06/04
Deadline
appendix
14/04
Appendix
validation
by
reviewers
2

7. Research Track
Only 3/13 papers
(+2 reacted too late)

8. See you at MMSys’18
Same process +
A reproducibility chair

9. A tradition of sharing
Tracks in all confs (MM, MMSys, ICMR)
Promotion of open source software (web)
Emblematic datasets
MediaEval benchmark, YFCC100m
… but not all papers are reproducible (yet)

10. Reviewers
Deal with exotic patched libraries
Reproducibility Chair
Collect artifact + metadata + licence
Organizers
Incentivize after acceptance
Challenges