Integration and Enhancement in Audiovisual Speech Perception

House Ear Institute, Los Angeles CA

Investigators

Abstract

With National Science Foundation support, Dr. Lynne E. Bernstein will conduct two years of research into the underlying brain mechanisms responsible for the human ability to combine speech information that is heard and speech information that is observed by watching a talker's face. The focus will be to explain the fact that being able to see a talker under noisy conditions dramatically improves the ability to hear that talker's speech. When measured, this effect is equivalent to almost quadrupling the loudness of the speech signal. A fundamental question is whether this effect occurs because listeners correlate speech information from the talker's lips and face with speech sounds, or whether the effect occurs whenever a visual object is paired with speech. Two main experiments will be done. The first will measure speech detectability levels in noise. It will compare perception of speech in noise with the same speech in noise paired with three types of stimuli: (1) a talking face, (2) a static but temporally aligned ellipse, (3) a dynamic ellipse whose vertical extent is controlled by the loudness of the speech signal. If the mere overlap between heard speech and a visual object results in improved perception, then the brain appears to turn up the gain when events overlap, regardless of whether they have similar significance. If the static ellipse is not effective but the dynamic one is, then the brain appears to depend on a correlation between stimuli but one that does not require the visual stimuli to be speech. But if only the talking face is effective, it is likely that the brain solves the noise problem by using processes that are specialized for speech. In the second experiment, electrophysiological recordings of brain activity will be made during a task like that in the first experiment. This experiment asks whether seeing a talker is the same as turning up the gain in the sound, as far as the brain is concerned. The event-related potentials to be obtained will be analyzed to find when brain events occurred and where in the brain they occurred. Other analyses will be used to investigate how the visual and auditory processing areas of the brain synchronize their activity during the detection of speech sounds. How the brain combines information from different sensory-perceptual modalities is one of the great mysteries of human perception, along with whether speech is processed by the brain in the same or a different manner than it processes other types of stimuli. This project is among the first to use both brain and behavioral methods to investigate how the brain combines auditory and visual speech under noisy conditions. Until recently, the techniques to study two senses at once were not available, and so little research investigated how the brain creates a coherent perception of the world from the diverse information it gets. Knowledge to be obtained will have practical implications. For example, it can suggest how visual stimuli can help listeners to get critical information under noisy conditions such as an airplane cockpit. It can help explain why people with hearing impairments benefit from being able to communicate face-to-face. The knowledge can also be extended to developmental research to determine why children have more difficulty than adults when listening to speech under noisy conditions, such as a noisy classroom.