Collaborative Research: RI: Small: Wisdom of Crowds with Machines in the Loop

Harvard University, Cambridge MA

Investigators

Abstract

The importance of both human and machine intelligence and their complementarity has given rise to the aspiration for human-machine hybrid computing systems that achieve more than either could alone. Human-in-the-loop computing, where human inputs are sought during the computation process, is a natural approach. However, most human-in-the-loop computing systems focus on how simple human inputs can help machines to better perform their tasks. This research takes the opposite perspective by focusing on a human-centered domain---the wisdom of crowds---and studies how having machines in the loop can improve the efficacy of harnessing the wisdom of crowds. A key challenge is directly evaluating the quality of crowd contributions. This research tackles the problem of obtaining high-quality contributions from the crowd despite the lack of data for such evaluations. This project seeks to make more accurate and robust use of crowd contributions in a broad spectrum of applications in business (e.g. crowd transcription and translation, and online reviews), sciences (e.g. citizen sciences, machine learning, and peer reviews for conferences and journals), education (e.g. peer grading) and other areas. This research investigates two core problems for tapping into the wisdom of crowds in the challenging, yet realistic, non-verification and unsupervised setting where no ground truth is available, addressing two key research questions: (1) how to elicit high-quality information from (potentially strategic) crowd members; and (2) how to aggregate the elicited information to form a high-quality, collective opinion. Lack of verification via ground truth presents a challenge for the mechanism designer to align incentives for elicitation. It also means that the designer does not know whose information should be weighted higher in aggregation given heterogeneous contributions. This research develops a theoretically grounded framework for elicitation and aggregation for settings without verification. It incorporates machine learning methods for the design of elicitation and aggregation mechanisms to achieve provable guarantees for the crowdsourcing applications, with a focus on the quality of elicited information and the quality of the aggregated opinion. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.