NSF PD 19-125Y - Science of Science: Discovery, Communication, and Impact (SoS:DCI) -Randomized Control Trial of the Registered Reports Publishing Format

Center For Open Science, Charlottesville VA

Investigators

Abstract

The Registered Reports (RR) publishing model could be transformative to how research is conducted, peer reviewed, and published. RRs could alter incentives for researchers toward focusing on asking important questions and using strong methodology to test those questions, and away from the demand for exciting, provocative results whatever their credibility. Ultimately, RRs can improve the quality of peer review and scholarly communication, thus improving the rigor, transparency, and reproducibility of research. Large-scale adoption of RRs could fundamentally change how research is evaluated and communicated. In a RR, authors submit a paper detailing the research question and proposed methodology to test the question prior to observing the study outcomes. In the first stage of peer review, reviewers assess the importance of the question and the quality of the methodology proposed to investigate it. If the paper passes this stage of peer review, the authors receive “in-principle acceptance” meaning that the journal commits to publishing the paper regardless of outcomes as long as the authors follow through with competent execution and reporting of the research. The second stage of peer review, after the results are known and added to the paper, assesses adherence to the original commitments, clarity of the distinction between planned and unplanned analyses, and accuracy of interpretation of findings, and not whether the results are positive, interesting, or consistent with hypotheses. Given the potentially transformative impact of RRs on the research process, it is essential to conduct a randomized trial to evaluate the model’s qualities. By collaborating with journals, we will conduct a randomized trial of RRs in an ecologically valid context. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.