Developing Methodology for Commensuration Bias Detection in Grant Application Peer Review

University Of Washington, Seattle WA

Investigators

Abstract

Through its six federal grant agencies, the United States invests billions of dollars annually to promote science, technology, and engineering research at colleges and universities. The long-term goal of this public investment of money and trust is to improve our nation's health, economy, and social policies. In order to determine which researchers and projects will receive funding, grant agencies employ a process of peer review in which expert researchers evaluate the merits of submitted proposals. This longstanding social technology -- of relying on expert evaluation to inform determinations of merit -- empowers grant agencies to make funding decisions based on a fuller understanding of the social and scientific excellence of each project. As such, it is critical for grant agencies to employ rigorous and fair peer review processes in order to recruit, retain, and fund the best minds. This project builds on a growing scientific literature that studies how grant peer review works with an eye towards identifying ways of improving its effectiveness. More specifically, grant proposal review procedures commonly require reviewers to score applications along multiple dimensions -- for example, a proposal's approach, innovation, versus significance -- as an intermediate step in determining the proposal's overall score. When procedures are left unspecified for how reviewers should combine individual scores (along multiple dimensions) into overall scores, evaluators might arrive at overall scores in ways that subtly advantage and disadvantage grant proposals submitted by applicants from different social groups. Any such difference is what we call commensuration bias. This research identifies and evaluates approaches for measuring commensuration bias by analyzing peer review data from applications submitted to an ongoing intramural collaborative biomedical research program that utilized the independent peer review services of the American Institute of Biological Sciences. This project aims to offer concrete, efficient policies that ensure fair review for any grant agency that requires scoring of applications along multiple criteria, including the National Institutes of Health, which is the world's largest public funder of biomedical research in the world. There is currently no established methodology for detecting commensuration bias. The availability of criteria and overall scores from individual reviewers makes it possible to examine how individual reviewers evaluate multiple criteria simultaneously and how they unconsciously combine criteria scores to arrive at overall scores. In addition to hierarchical linear models that assume overall application score is an additive function of criteria scores, this study uses Bayesian regression trees to model non-linear decision processes and multivariate analyses to model the distribution of criteria scores. This range of statistical approaches allows this study to avoid making strong assumptions about the nature of commensuration and provides tools needed to inform two main types of potential policy recommendations: (a) those that focus on the restructuring or modification of programmatic review procedures as a whole and (b) those that focus on the monitoring of unusual peer review scores. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.