Assessing Bias and Idiosyncrasies in Elite Scientific Peer Review

University Of Colorado At Boulder, Boulder CO

Investigators

Abstract

Peer review is a core process by which the scientific community formally evaluates new research contributions. The outcomes of peer review for scientific articles, particularly at elite scientific journals, have broad influence on the public’s understanding of scientific knowledge, as well as on scientific careers and the directions of scientific discovery. At the same time, peer review can exhibit biases and idiosyncrasies that can produce non-meritocratic outcomes, which can, in turn, limit scientific advancement and broad participation, and can even undermine or impede evidence-based policy. This project will (i) quantify, model, and understand the magnitude, sources, and effects of biases and idiosyncrasies within elite scientific peer review across multiple scientific fields, (ii) facilitate broader community efforts in studying peer review using scientific methods, and (iii) inform new policies intended to ensure the reliability of peer review and its outcomes. This project will (i) produce multiple comprehensive, anonymized datasets of peer review at two elite general science journals, spanning multiple years and fields, and make them publicly available for reuse by the research community; and (ii) use these anonymized datasets to quantitatively assess the magnitude, source, and effects of both social biases and editorial idiosyncrasies within elite peer review. These analyses will use statistical methods from causal inference, statistical modeling, machine learning, and natural language processing to produce state-of-the-art estimates of effect sizes and sources. The anonymized datasets will adhere to the NIST IR 8053 standard for de-identification, and the anonymization scheme will be robustly evaluated prior to any data release. These data sets will enable quantitative, longitudinal analyses of submitted manuscripts, and the dependence of different editorial and reviewer recommendations on social demographic and professional attribute variables, to statistically assess, quantify, and compare social biases and editorial idiosyncrasies across manuscripts, editors, and fields. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.