CAREER: Fundamentals of Learning from People with Applications to Peer Review

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

In a wide variety of applications -- such as peer review, recommender systems, hiring, college admissions, peer grading, A/B testing, and crowdsourcing -- it is common to elicit and process data from people. Such data often suffer from issues such as miscalibration, subjectivity, strategic behavior, and biases. These issues are further amplified in applications (such as those listed above) in which the data comprises evaluations of a set of items by people, where every person evaluates only a subset of items and every item is evaluated by only a subset of people. These issues degrade the overall quality of these applications and also lead to unfairness towards some of its participants. For example, data from people often have biases pertaining to certain demographics; subjective opinions or strictness/leniency of the human evaluators can lead to unfairness; some participants may indulge in strategic behavior which can be detrimental to the overall system. This project will design algorithms for eliciting data from people and processing it in a manner that mitigates these issues to the maximum possible extent. The project will have a particular focus on the application of peer review of scholarly research. It will make a significant real-world impact through the research outcomes for applications that depend on data from people, outreach to drive positive policy changes, and synergistic educational activities. This project will address the issues of miscalibration, subjectivity, strategic behavior and biases in learning from people along three fronts. First, using tools from information theory and statistics, it will establish the fundamental limits on the extent to which these problems can be mitigated. Second, it will develop algorithms that will provably achieve (or approach) these limits, and are also computationally efficient. The research on this front will employ tools from machine learning and statistics, game theory and social choice theory. Finally, the project will transform the theory into a useful toolkit for practitioners, as well as outreach towards driving positive policy changes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.