The Multisensory Training Benefit for Speech and Speaker Perception

University Of California-Riverside, Riverside CA

Investigators

Abstract

When listening to someone in a noisy environment, you can better understand what they say when you look at their face while they talk. Providing audiovisual information about speech, not just audio information alone, has also been shown to help infants develop language. In some cases, using audiovisual information to train speech perception helps adults learn a second language, helps aphasic patients produce more fluent speech, and improves perception of heard speech by listeners with hearing impairments. This last group is of particular interest because roughly a third of U.S. adults have some degree of hearing impairment. While this audiovisual training advantage is well established, it is unclear how exactly the effect works. Research will be conducted to determine the principles behind the effect, how and why it works, and in what circumstances. The investigators also explore why audiovisual training typically does not help hearing-impaired individuals who received cochlear implants after early childhood. One possibility is that their years of experience before receiving the cochlear implant required a heavy reliance on lip reading, leaving them over-reliant on information from the talker during audiovisual training. The specific issues examined are rooted in theories of multisensory learning. These issues include: a) whether simultaneous auditory and visual speech is necessary for the training advantage to occur; b) to what degree the advantage generalizes across different talkers for a given listener; and c) whether the use of other sensory channels, in this case audio and touch, results in a similar training advantage. Parallel experiments will be carried out with both normal-hearing individuals and individuals who received cochlear implants after early childhood. The research should not only improve speech perception training regimes for individuals in this clinical population but should also improve our understanding of typical speech and talker perception. More generally, the research should inform our understanding of perceptual learning and plasticity, as well as multisensory integration.