Mapping Affect and Facial Dynamics during Dyadic Conversation

University Of Virginia Main Campus, Charlottesville VA

Investigators

Abstract

As we talk with others, facial expressions, head movements and nonverbal speech characteristics serve as crucial cues to our emotional tone. For this process to be effective, the cues to expressed affect must change constantly and are difficult to study. The lack of tools for measuring and modeling the multiple cues used to interpret live facial expressions has limited the scientific study of the regulation of facial expression in conversation. This proposal consists of experiments designed to manipulate the specific cues thought to determine perceivers' understanding of communicators' emotional states during live conversations. The primary goals of this project are twofold. The first goal is to develop and refine tools (e.g., computer software programs) that will allow scientists to better understand the specific facial and movement cues that allow people to express and understand communicated affect during live conversation. The second goal is to refine the software that allows such tests of affective communication, so that it serves as a new tool for a host of behavioral scientists who wish to study and manipulate the nature of live interactions. For example, this software will allow experimenters to manipulate the perceived gender of two people who are having a live getting-acquainted conversation -- so that a man who is actually talking to another man sees a realistic, convincing facial avatar of a woman (who perfectly mimics the facial and head movements of the actual male interaction partner). This software developed and refined in this proposal will thus allow social and cognitive psychologists to manipulate, for example, the social categories of people's interaction partners in ways not imagined a decade ago. The impact of this project is exceptionally wide ranging. One area of application is measuring and modeling facial dynamics -- that is, using this technology to further study live facial expression. The current project will provide open source software that can be distributed free for research purposes. Research labs who wish to use the software will also be provided with free supporting materials (e.g., plans, equipment lists) to facilitate the dissemination of the technology resulting from the current project. A second area of application is the study of intercultural communication. This new technology will allow sophisticated studies of social interactions and communication in small group, high stress settings in which emotional regulation is critical (e.g., intercultural interactions by police or military personnel) or in negotiation settings (e.g., diplomatic relations). Further the technology could be used to allow live video interactions between people that maintain the tone and content of communications while still maintaining speaker confidentiality. A third area of application is educational technology. A problem with virtual learning environments is the difficulty of detecting students' nonverbal cues -- cues that a live classroom teacher often uses to assess student understanding or confusion. The work may lead to automatic recognition of these cues from video capture. This information could then be used to improve the virtual learning environment or refine the specific teaching materials that are broadcast to students.