EAGER: Policy Design for Reproducibility and Data Sharing in Computational Science

Columbia University, New York NY

Investigators

Abstract

Scientific computation is emerging as absolutely central to the scientific method, but the prevalence of very relaxed practices regarding the communication of experimental details and the validation of results is leading to a credibility crisis. Across the computational sciences, the research community is now questioning traditional modes of communication and seeking to implement methods for scientific knowledge transfer that make replication of published computational findings possible. This typically means making all details of the computations - the data and code - underlying published computational results conveniently available to others. Making data and code openly available raises myriad questions regarding appropriate and effective ways of reaching the goal of reproducible research. The requirements of scientific journals exert a powerful influence on publishing decisions, and are also the least well-understood. This proposal seeks to build an understanding of the current state of journal policy regarding reproducibility of published computational results, and of the factors underlying journal policy changes toward the adoption of data and code sharing. The highly granular nature of computational science research provides a natural experiment for the study of effectiveness of policies in communities with different attendant pressures such as data and codebase size, privacy and legal barriers, capital intensity and level of instrumentation, and industrial collaboration, to name a few. Measures of effectiveness can be ascertained since domain-specific journals show a spectrum of positions on data and code sharing, from requiring both to ignoring the issue altogether. These findings will in turn inform the creation of guidelines regarding effective data and code sharing policies across the computational research landscape. In addition, this research will conduct detailed case studies describing successful approaches journals have used to further reproducible research. Finally, this project will also act as a case study itself, with the open release of the data and code underlying its published results and analysis of how best to facilitate reproducibility for similarly situated research.