Cross-Modal Information for Speech and Speakers

University Of California-Riverside, Riverside CA

Investigators

Abstract

This research will examine the relationship between speech and speaker perception as it exists across auditory and visual modalities. Recent results have shown that familiarity with a speaker can enhance speech recognition for both auditory speech perception and lipreading. Familiarity with a speaker's voice facilitates auditory speech recognition, and familiarity with a speaker's face facilitates lipreading. These findings challenge traditional theories, which have assumed independence between voice and speech perception, as well as between face recognition and lipreading. Current explanations of speaker-speech facilitation in both modalities have focused on information that is tied to each individual sense. Alternatively, the facilitation could be based on familiarity with a speaker's style of articulation, which is conveyed in both auditory and visual speech information. If the link between speech and speaker properties is based on this modality-neutral articulatory information, then speaker facilitation of speech perception should work across, as well as within, auditory and visual modalities. Three sets of experiments will be conducted to test this hypothesis. The first set will examine whether articulatory information can be used to identify speakers across auditory and visual domains. Experiments will test whether speakers' voices can be matched to their faces based on isolated articulatory information. The second set of experiments will test whether familiarization with a speaker in one modality facilitates recognition of that speaker's speech in the other modality. The final set of experiments will examine the relative influences of switching sensory modality and switching speakers within and between speech utterances. The results of this research should be illuminating about theories of speech and face perception, as well as general issues of multimodal integration. The research will address issues relevant to individuals with hearing impairments, as well as aphasic, prosopagnosic, and phonagnosic patients.