Visual Contributions to Speech Perception

Queen'S University At Kingston, Kingston ON

Investigators

Linked publications & trials

Abstract

[unreadable] DESCRIPTION (provided by applicant): It is well known that the intelligibility of auditory speech perception is enhanced when a listener can view the talker's face. This fact is important for developments in automatic recognition of speech as well for clinical intervention for individuals with hearing impairments. The proposed research addresses the nature of this visual information and what visual information processing takes place during audiovisual speech perception. The experimental strategy is to manipulate image characteristics systematically in order to uncover the necessary and sufficient visual conditions for speech intelligibility and to investigate how eye movement behavior is influenced by these characteristics. One important tool for these purposes is realistic facial animation and two different systems of facial animation are employed in the proposed research. Three main research aims are addressed: 1. The spatial resolution required to perceive visual speech will be assessed. In a series of studies, digital image processing is used to produce image sequences with restricted ranges of spatial frequencies. Differences in the perception of speech in noise with these image sequences will be used to determine how fine grained visual speech information must be. 2. The temporal characteristics of visual speech information will be assessed through studies using temporal filtering and animated talking heads. With the aid of realistic animation, the precise movement of the facial surface and head can be manipulated to test for perceptual sensitivity to movement parameters. 3. Eye movements will be monitored in studies in which image characteristics and visual speech kinematics are manipulated. The studies will contribute to the understanding of how visual stimulus properties and higher level processes determine gaze patterns and thus influence perception.