Synthesis and Evaluation of Visible Speech

University Of California-Santa Cruz, Santa Cruz CA

Investigators

Abstract

This research involves development, implementation, and evaluation of a computer animated talking head, which generates realistic visible speech coordinated with auditory speech. Synthetic visible speech has an obvious potential for advancing knowledge about speech production, the visible information in face-to-face speech perception, how it is utilized by human perceivers, and how it can best be used in communicating with individuals with hearing loss and in language learning situations. Realistic speech is obtained by animating the appropriate facial targets for each segment of speech along with the appropriate coarticulation, paralinguistic content, and emotion. The research goal is to achieve true photo-realism (visual speech indistinguishable from a real talker) or even superrealism (visual speech easier to read than a real talker). To achieve this goal, it is necessary to obtain additional physical measurements from real speech, to refine the control of the talking head, and to evaluate these changes using intelligibility testing with human observers and by comparisons to natural speech. The working hypotheses of this research are 1) a synthetic talker is an important challenge to speech research and computer animation and offers a potentially valuable medium for communication among both normal and disabled individuals, human-computer interaction, and virtual worlds, 2) synthetic visual speech will provide a valuable experimental tool for better understanding of speech perception by ear and by eye, 3) visual speech information offers an additional source of information for both normal and hearing-impaired individuals, and 4) the research will have immediate and direct application to improving the communication alternatives in noisy environments and for individuals with hearing loss.