CHS: Small: Scientific Design of Interactive Human Computation Systems

Georgia Tech Research Corporation, Atlanta GA

Investigators

Abstract

Human computation systems are intelligent systems that organize humans to manually carry out computational processes that are too hard for current computational systems to solve, collect commonsense knowledge typically not available automatically, or label data. Designing them to serve recreational functions promises to overcome the need for worker monetary compensation, incentivizing crowd workers to generate data and solutions in exchange for non-monetary rewards. However, early attempts have generally not lived up to their potential, in large part because we do not understand how design decisions will impact human worker performance and engagement. This project will develop a science of design for human computation systems, in three ways: (1) Controlled experiments on how mechanics common to modern commercial social and mobile applications can be adapted to human computation. (2) An intelligent system that can automatically conduct large-scale experiments on the effects of different system designs on human workers. (3) Investigation of how the design knowledge automatically learned by the system can be used to support human software designers via intelligent tools. This research will result in (a) novel human computation design patterns, (b) new methodologies for studying the constituent parts of human computation systems, (c) new understanding of how those patterns affect human crowd worker performance and engagement, and (d) new approaches to the development of design tools that incorporate intelligent creativity support. An improved comprehension of the effects of design considerations on users will make it easier and quicker to design, develop, and deploy effective systems. The research will develop optimal experimental design algorithms that automatically generate variations of human computation systems and conduct tests of their effects on human users, resulting in a Bayesian model of the effects of software mechanics on crowd worker performance and engagement.