RI: Small: Decision-Theoretic Control of Crowd-Sourced Workflows

University Of Washington, Seattle WA

Investigators

Abstract

Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people as an open request for services. Requesters use crowd-sourcing for a wide variety of jobs like dictation-transcription, content screening, linguistic tasks, user-studies, etc. These requesters often use complex workflows to subdivide a large task into bite-sized pieces (including the management of these tasks), each of which is independently crowd-sourced. These workflows are paramount to the success of crowd-sourcing, still, there has been little attention paid to methods for dynamically optimizing the throughput of a workflow. Controlling and optimizing such a workflow is an excellent application for AI research for two reasons. First, it is challenging in that the agent has to understand the dynamics of an uncertain, real-time environment and reason about distinct choices for a decision. More importantly, the domain has significant economic value -- progress can potentially impact hundreds of thousands of people and spur economic development in a fast growing sector. This project is investigating complex workflows using a decision-theoretic framework that optimizes for a quality/price trade-off, with aims of (1) building statistical models of worker behavior derived from a large corpus of online behavior, (2) defining a declarative representation language to describe a wide range of workflows, and (3) developing an automated scheme that optimizes a general workflow resulting in an automated controller for making informed decisions at various stages of the process and for monitoring worker accuracies and computing corrections based on them. In the longer term, perhaps beyond the scope of this project, is (4) development of an interface optimizer that automatically learns the best user interface for a task based on user behavior increasing throughput of the workflow, and (5) integrating these ideas in an open-source, software toolkit to directly benefit the various requesters in managing their tasks.