The Temporal Organization of Prosody in Multimodal Speech and Co-speech Gestures

Lee, Yoonjeong, Los Angeles CA

Investigators

Abstract

This award was provided as part of NSF’s Social, Behavioral and Economic Sciences Postdoctoral Research Fellowships (SPRF) program. The goal of the SPRF program is to prepare promising, early career doctoral-level scientists for scientific careers in academia, industry or private sector, and government. SPRF awards involve two years of training under the sponsorship of established scientists and encourage Postdoctoral Fellows to perform independent research. NSF seeks to promote the participation of scientists from all segments of the scientific community, including those from underrepresented groups, in its research programs and activities; the postdoctoral period is considered to be an important level of professional development in attaining this goal. Each Postdoctoral Fellow must address important scientific questions that advance their respective disciplinary fields. Under the sponsorship of Dr. Jelena Krivokapić at the University of Michigan, this postdoctoral fellowship award supports an early career scientist examining the nature of coordination between multimodal speech events. In everyday communication, speech and gestures that accompany speech, referred to as co-speech gestures, interact in a principled way. While much of interdisciplinary research has examined the multifaceted nature of verbal and non-verbal behaviors in human communication, our understanding of the multimodal prosodic phenomena remains limited. The goal of the proposed research is to illuminate how language users integrate prosody into different modalities while producing words and phrases. Prosodic structure groups words into larger phrases and expresses prominence (highlighting important or new information). Specifically, this project examines how prosodic information is embodied in co-speech gestures occurring at phrasal edges and under prominence, what coordination patterns of speech and co-speech gestures emerge from the prosodic structure that is specific to a language, and if and how interacting language users exhibit prosodic accommodation in co-speech gestures. By using cutting-edge instruments and analysis tools, multimodal data (pitch, movement of the vocal tract, head, eyebrow, hand) are assessed via concurrent recordings of audio, video, and kinematic information from individual speakers of typologically different languages (English and Korean) participating in various speech tasks including solo and interactive speech activities. The proposed research examines how the speech signal and the accompanying bodily movements produced by a language user are mediated by prosodic structure of the language. This project utilizes cutting-edge techniques to collect the necessary multimodal data and characterizes the temporal relation between co-speech beat gestures, pitch gestures, and consonant and vowel gestures as well as detailing their kinematic properties at phrasal boundaries and under prominence. Further, a series of multimodal experiments is designed for two prosodically different languages—a language with no lexical stress and no pitch accent and a language with stress and marking prominence with pitch accents to reveal the cross-linguistic and/or language-specific aspects of speech and co-speech gesture coordination. This research also conducts a real-time accommodation study designed for pairs of speakers participating in communicative speech tasks, with the broad goal of gaining information about the emergence of prosodic accommodation across modalities. To date, the proposed work is the first to investigate accommodation processes in both speech and co-speech gestures. This work can advance our understanding of the precise nature of the speech and co-speech gesture coordination and of how different modalities are coordinated to embody information grouping and highlighting at the phrasal level, broadening the empirical and theoretical base for models of articulatory organization and linguistic accounts of prosody. The results serve to support the development of theoretical models of prosodic structure of typologically different languages that take into account these different modalities of communication. The proposed work produces a substantial database of the movement signals of multimodal articulators with synchronous audio and video recordings that will be freely distributed to the research and education community. Broadly, this research contributes to knowledge and perspectives for future exploration of multimodality in communication in sign language, education, and engineering and clinical applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.