Biocomplexity Incubation Activity: Coordination of Perception and Spoken Language Processes

House Ear Institute, Los Angeles CA

Investigators

Abstract

Human speech processing engages multiple neural resources, exhibits individual neural reorganization and variation in response to system perturbations/disorders, evolves along and is also constrained by temporal processes, and responds to statistical properties in the language environment. Human speech perception comprises not only auditory perceptual processes but also visual ones. For example, seeing a talker enhances the ability to accurately perceive speech in noise. Auditory and visual processing of the speech signal is a biocomplex phenomenon. During audiovisual speech perception, information is integrated across perceptual systems to give rise to unitary speech percepts. This integration implies the coordination of perceptual, psycholinguistic, and possibly cognitive systems such as attention. Perception emerges from processing at the levels of cortical and sub-cortical brain structures, but perception cannot be reduced simply to its underlying neural mechanisms, it must also be accounted for in behavioral terms. Speech perception emerges across temporal scales, including the scale of development and the scales of communication. Speech perception also is responsive to environmental demands, as is evident in the development of perceptual skills by expert deaf lipreaders. This Biocomplexity Incubation Activity award will support a project that seeks to create a development context for researchers who might solve the problem of how the auditory and visual perceptual systems are integrated with the spoken language system, but who have not yet had the opportunity to learn enough about each other's fields and perspectives to propose a research program. Researchers with expertise in visual psychophysics and perception, multimodal integration, auditory and visual speech perception, neuroanatomy and neurophysiology, and computational modeling will meet in a regularly scheduled research seminar covering visual and auditory perception in relation to spoken language processing. The seminar will include demonstrations of relevant phenomena and experimental methods/apparatus. Demonstrations for the seminar are intended to teach and also generate discussion. They will include examples such as the McGurk effect, sinewave speech, point light speech, random-dot motion effects, structure from motion and others. The research seminar will be assisted by an undergraduate student who will help to create the demonstrations. These demonstrations will be made available on the House Ear Institute (HEI) Web site. The seminar will develop hypotheses and methods to form the basis for future collaborative research. The foci will be auditory perception, visual perception, audiovisual integration, and visual speech perception enhancement. A workshop will conclude the research seminar. Its purposes are to present the generated hypotheses and methods and to obtain feedback. Presentations will outline the biocomplexity issues addressed in the seminar, the key lines of research identified to be of critical importance, and the experimental and computational approaches following those lines. Expert panelists from other Southern California institutions will participate in the workshop. A final report on the project and the workshop will be prepared for the HEI Web site. The use of spoken language is a fundamental distinguishing characteristic of humans, but knowledge is incomplete regarding how humans accomplish it. The particular focus of this project is to understand how the perceptual systems of vision and audition process linguistically relevant information. A complete understanding of perception and language will need to comprise links between levels of explanation, from the neural to the behavioral. Societal use of this fundamental knowledge will be in such areas as that of enhancing learning and teaching, ameliorating the effects of diseases such as stroke and impairments such as hearing loss, and developing artificial systems for processing language (e.g., automatic speech recognition and machine language translation).