Listening effort and audiovisual speech by L1 ("native") and LX ("nonnative") English speakers

Carleton College, Northfield MN

Investigators

Linked publications, trials & patents

Abstract

Project summary One of the most robust findings in the speech literature is that listeners can understand speech more successfully when they can see as well as hear the talker. However, there is considerable debate about whether seeing a talker’s face increases or decreases the cognitive and attentional requirements for successfully processing speech (“listening effort”), relative to hearing alone. Conflicting findings in the research may be due, in part, to a lack of systematic evaluation of multiple factors that moderate the effects of visual input on listening effort. The proposed research, which will involve undergraduate researchers at every step of the process, will clarify how audiovisual speech affects multiple facets of listening effort and will assess whether and how these findings are moderated by the difficulty of the speech task. Further, it will assess whether these findings extend to participants who have historically been excluded from speech research: those for whom English is not their first language. The studies will: 1) Evaluate the consequences of audiovisual speech on listening effort, as quantified by three commonly-used measures of effort: subjective effort, dual-task costs, and recall. 2) Assess the moderating effect of signal-to-noise ratio on the effort associated with processing audio-only and audiovisual speech. 3) Evaluate how the language background of the listeners affects the effort associated with processing audio-only and audiovisual speech. This will be the first study to assess whether and how the listening effort associated with processing audiovisual speech differs for people who learned English before any other language and those who learned another language before English. Given the ubiquity of audiovisual speech and the fact that maintaining high levels of listening effort can lead to mental fatigue or distress, understanding how visual information affects listening effort has important practical and clinical consequences.