Understanding Multi-Modal Discourse: Cognitive Resources and Speech-Gesture Integration

University Of California-San Diego, La Jolla CA

Investigators

Abstract

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5. Anyone who has had a phone conversation about how to how to change the oil in a car or how to make lasagna knows the importance of explanatory gestures for communication. Multimodal discourse involves the use of both visual (gestures) and auditory (speech) information during communication. Scientists still have much to learn about how people combine information in gestures with information in speech and whether people vary in their ability to use gestural information. This project evaluates the importance of working memory for combining linguistic information in speech with gestural information. Participant's brain activity will be recorded as they watch videotapes of a person talking about concrete topics such as the shape of objects, their relative sizes, and other spatial information that is difficult to convey through speech alone. The impact of gestural information will be measured by comparing brain activity recorded when the same person watches videos in which the speaker does gesture with those in which he does not. The importance of the different working memory systems will be assessed by seeing how language comprehension suffers when participants are asked to remember irrelevant words (verbal working memory), dot patterns (visuospatial working memory), and pictures of meaningless body positions (sensorimotor working memory). The impact of differences in learning style and cognitive abilities on gesture comprehension will also be examined By discovering the relative importance of verbal, visuospatial, and sensorimotor working memory for understanding face-to-face communication, this research will aid the design of more effective teaching and training methods. The project may guide the development of teaching methods that are specially adapted for people with different learning styles and could help children and adults with communicative deficits by maximizing the effects of gestural information. The project supports an early career scientist and provides summer jobs for college students, including those from minorities under-represented in science.