SoCS: Effectively Leveraging Contributions in Human Computation Systems

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

Human computation studies how to collect useful data as a by-product of another activity in which people are interested (e.g., playing games). A popular example is the ESP Game, where two players are shown the same image and must independently generate tags; tags that match become labels for the image. ESP Game players have generated millions of labels that help improve image search engines. Currently, little is understood about how to capitalize on each person's individual expertise to produce the best results in human computation systems. For example, the ESP Game could generate better results if automotive enthusiasts labeled images of cars while biologists labeled images of animals. This project aims to better understand how each individual's different capabilities can be assessed, dynamically leveraged, and even improved for the purposes of human-driven data collection. Intellectual Merit: Improved understanding of the strengths and weaknesses of human users as teachers and data sources; an intelligent new objective-driven model of data collection; novel opportunities to study machine learning algorithms that capitalize on human teachers' abilities; and an analysis of learning opportunities as incentives for people to participate in human computation systems. Broader Impact: Distribution of large new data sets (e.g., Wikipedia articles in multiple languages); several Internet-based human computation systems for large-scale evaluation of machine learning and other algorithms; a new course called ?Human-in-the-Loop Systems?; workshops held in conjunction with major conferences; and outreach activities (e.g., summer projects) that introduce female undergraduate students to interdisciplinary research.