EAGER: An Animated Agent for Developing the Conversational Skills of Individuals with Social Interaction Difficulties

University Of Rochester, Rochester NY

Investigators

Abstract

This EArly Grant for Exploratory Research seeks to develop an exploratory version of a computer-based animated agent that simulates a conversational partner, allowing users with social interaction difficulties, such as ones with Asperger syndrome, to practice conversational skills. The interaction will explore establishing friendly rapport with the user through conversation; building such rapport is an important social skill that is difficult for the targeted individuals to acquire. It is hypothesized that such individuals will very significantly benefit from the exploratory agent, by allowing them to practice conversational skills without risk of failure or of making a poor impression. The most exciting part of this exploratory technology is that it puts users in the driver's seat of the interaction with a standardized, objective and repeatable stimulus. The architectural framework for the "getting acquainted" scenario is general enough to be adaptable to many other restricted conversational scenarios of societal importance such as job interviews, customer service, and participating in group decision-making. The exploratory agent is a simulated adult responding in the mature, patient manner of an experienced consultant (an expert advisor from the University of Rochester Medical Center consults on the project). The agent correlates the topic and content of the user's utterances with the prosody, facial expressions, gaze direction and head motions accompanying the utterances. The agent also provides helpful real-time feedback via red/green screen icons about these behaviors, and behavioral summaries and suggestions for improving the behavior at the end of a session. The initial domain in this exploratory project is "getting acquainted", i.e., engaging in the sort of small talk that is conventional for individuals who have not previously met. The dialogue framework has some transactional knowledge, independent of topic, to initiate a conversation and to respond appropriately during a conversation. When the user's responses to the agent's questions and prompts deviate from expectations, the agent branches to clusters of strategies including evasive answers, requesting repetition, providing encouragement and prompting further input. The project includes exploration of more general questions such as the architecture required for integrating nonlinguistic and linguistic behavior in an animated agent. The project also leads to new user interfaces that can capture nonverbal behavior analytics reflecting the dynamics of the conversation.