EAGER: Can Data Mining and Crowd Sourcing Revolutionize the Study of Scientific Peer Review? Generating a National, Open Depository of Grant Review Outcomes from Federal Agencies

University Of Wisconsin-Madison, Madison WI

Investigators

Abstract

One major barrier to research on peer review is the inaccessibility of data. Federal agencies are required to protect the identities of reviewers and applicants, which makes sharing data challenging, and prevents agencies from capitalizing on the opportunity to improve their systems with the help of the scientific community. This project addresses this problem with a novel automated program that collects peer review outcomes, such as grant application critiques and scores, directly from applicants, and de-identifies and stores them in a large repository that the scientific community can access. More intensive study and refinement of federal agencies' review processes will help to inform federal policies to ensure that U.S. tax dollars are allocated in ways that yield the broadest benefit. Conducting this work will also help broaden participation in science because it has high potential to illuminate reasons why individuals from historically underrepresented groups face disadvantage in peer review processes. Beginning with the National Institutes of Health (NIH), the largest federal funder of biomedical research, this project's automated data collection program uses web crawlers to identify Principal Investigators (PIs) and their email addresses from NIH's public access database, RePORTER; the program then sends PIs email invitations to donate their review materials (i.e., critiques and scores), which are de-identified and placed in a protected database (with the option to "opt-out" for all PIs). The program asks participating PIs to provide demographic information, and merges demographic and de-identified review outcome information into datasets that can be used for data and text analysis. This automated program can be adapted to collect review outcomes from unfunded applications, as well as from other federal funding agencies, by targeting broader email lists.