CAREER: Continual Automated Refinement of Human Computation Systems

Northeastern University, Boston MA

Investigators

Abstract

This research aims to improve automated tools for the application of human and computational problem-solving systems. This will lead to generalized techniques for data-driven modeling and optimization of the process of designing such systems, reducing the workload necessary to create successful ones and broadening the scope of problem domains to which massive amounts of human brainpower can be applied. Despite the vast computational power currently available, a broad range of important problems still rely on human reasoning or intuition to solve. In cases where algorithms are either unknown or computationally intractable, human computation has recently arisen as a means to apply human skills to advance solutions to problems neither humans nor computers could solve alone. By bringing human creativity, problem solving, and perspective to bear, humans and computers combined can solve previously unsolvable problems. Additionally, these systems create a new pathway for involvement in science - a new way for people to contribute towards problems that are important to them. By democratizing science, we involve those who may not otherwise have had such a means. Finally, this research can contribute to our understanding of how to best train people in solving challenging problems. This work seeks to automate one aspect of the iterative refinement of human computation systems: improving the assignment of tasks to contributors. The basic approach is to construct a model of contributors and tasks, based on skill ratings and skill chains, which can be used to assign contributors an appropriate task to complete. This model will automatically refine the skill estimates and assignments over time based on data, improving both user experience and problem solving outcomes. This approach in broken down into three challenge areas: 1) developing a unified skill model that combines skill atoms and skill ratings, then using that skill model for 2) crafting a difficulty curve tailored for each participant, and 3) evaluating design decisions. The approach will build on existing multi-person matchmaking systems, validated in multiple human computation systems.