Collaborative Research: Comparing Single- vs. Double-Blind Review of Scientific Abstracts for Accuracy and Bias

Dartmouth College, Hanover NH

Investigators

Abstract

This research project will conduct a field experiment to compare double-blind and single-blind review processes in a high-stakes selection process. Peer-review is used to identify quality scientific work and as such is the mechanism by which the scientific community regulates itself. Science, however, does not take place in a vacuum. The possibility of bias from known characteristics such as gender, name, title, or nationality has led to frequent and recurrent calls that the evaluation of scientific work should follow a double-blind review process (where the identities of both the reviewers and the authors are withheld) rather than a single-blind review process (where the identities of the reviewer are withheld but not the authors). Yet, there are strong arguments for the use of single-blind review, including that it is difficult to truly conceal the identity of the authors and that their identities may serve as a valid cue for the quality of the work. The project will compare the outcomes from single- and double-blind reviews of talks included on the program of a national conference. The experiment will examine how well each review process predicts highly rated talks (accuracy) and whether the reviewers show preferences for author characteristics unrelated to talk quality (bias). The project will provide evidence-based recommendations for review procedures that facilitate an efficient, fair, and impartial review process. The project also will help develop methods to insure underrepresented groups have an equal and fair chance in other fields where peer review plays an important role in the research selection process, such as science, technology, engineering, and math. Graduate students will play an integral role in the conduct of this research. This research project will compare the effects of single- and double-blind review processes on conference submissions to the upcoming 2018 Annual Meeting of the Society of Judgment and Decision Making in a large-scale field study. The project will examine the degree to which characteristics such as gender, name, title, country of origin, and institution influence evaluation of conference abstracts in a high-stakes selection process. The project also will examine the impact of the use of these cues on evaluation accuracy. All abstracts submitted for a paper presentation will be subject to both single- and double-blind review. This will allow a direct comparison of the two review processes controlling for heterogeneity between papers. The accepted talks at the annual conference also will be evaluated for their quality and attendance to determine which of the two review methods better predicts these outcomes. No published study has compared actual outcomes of different review processes. Reviewer ratings will be modeled not only using the characteristics of the submitting author, but also the features of the topics in the submitted abstract, thus allowing for comparison of how different review processes impact the judgment processes of reviewers. The investigators will use the results from this large-scale field experiment to develop evidence-based recommendations for review procedures that facilitate an efficient, fair, and impartial review process. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.