RI: Small: Robustness to Undesirable Behavior in Peer Review

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Peer review is the backbone of scientific research. It is used widely to review the quality of papers as well as for awarding grants worth billions of dollars every year. However, due to more quantitative evaluations for jobs and promotions, incentives to have papers or proposals accepted are only growing. While the vast majority of the research community consists of honest people, recently there have been discoveries of a certain kind of fraud in peer review. In this type of fraud, a group of authors and reviewers form a coalition outside of the review process, and aim to get assigned each others' papers or proposals and then illegitimately accept them. If left untreated, there is significant risk of bad behavior spreading. The presence and prevalence of such malicious activities also leads to increased skepticism and distrust in science among the general public. The proposed research will have a broad impact in terms of addressing these critical issues. This research will develop a computational toolkit for organizers of peer review to detect and/or mitigate such fraud. The researchers will also assist any organizers who wish to employ this toolkit. The large number of submissions in many conferences and the increase in scale and sophistication of manipulation attempts makes it infeasible to manually detect and/or mitigate these issues. This research will design a theoretical framework that explicitly considers coalition-based fraud, and allows to make precise claims about guarantees that the algorithms give us. The research will also design algorithms that achieve desirable guarantees within that framework. These will be built into an open-sourced toolkit that will be made publicly available for use. The research will design and use techniques spanning machine learning, optimization, statistics, and game theory and mechanism design. The research will advance the understanding of the fundamental limits of mitigating collusions in various peer-assessment settings, and how these limits can be achieved or approached through the design of scalable and practical algorithms. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.