ITR: Multimodal Human Computer Interaction: Toward a Proactive Computer

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

This is the first year funding of a five-year continuing award. This project is based on the belief that, to be more accessible to the general population, computers must be more proactive in their interactions with people. In human interaction, someone who waits for each command before making any communication attempt would be regarded as uncooperative and unhelpful. In order for a computer to be more proactive and, thus, to bear its part of the burden of initiation in interactions, it must have (1) much more real-time information about its user, and (2) algorithms that select actions based on this information rather than simply on user commands. The computer needs information about the user's current and past emotional, motivational and cognitive state as well as the state of the task at hand. A theory, is needed to guide the development of algorithms that select appropriate actions based on user and task state. This research constitutes the next steps in an attempt by the PI's multidisciplinary team to develop this capability. Proposed research includes: (1) further development of methods to sense user postures, movements, expressions and speech; (2) analysis and fusion of this information to identify and track user states; (3) task state tracking; (4) creating a corpus of emotion- and action-labeled videotapes for use with computer learning; (5) further development of affective communication; (6) development of the basis for human-centered state-based action decisions; and (7) evaluation of computer proaction on human behavior and response. The testbed is an environment for hands-on education in science and engineering, using the Lego Mindstorms construction and robotics environment, with children of middle school age. An emphasis will be on developing proactive computing methods for encouraging interest and conceptual development of minority children and females, who often show lower achievement in science. Although the work will be conducted within an educational environment, the methods developed and studied will be broadly applicable, and this project should serve as an exemplar of the type of work that is needed in other computer-aided situations.