CRCNS US-Spain Research Proposal: Collaborative Research: Tracking and modeling the neurobiology of multilingual speech recognition

University Of Connecticut, Storrs CT

Investigators

Abstract

More than half of the world's population speaks two or more languages fluently. Speaking more than one language allows you to communicate and interact with individuals in other countries and cultures through conversation and reading. It also has economic benefits, and possibly even cognitive and health benefits. Scientifically, there are many open questions about bilingualism for education (how can we best train people in new languages?), health (how can you best treat a bilingual person with language difficulty after a brain injury?), and technology (how can we make speech recognition systems as flexible as multilingual humans?). To better understand multilingual language processing, researchers will record neural responses to speech in people who know either one, two, or three languages, while they listen to the languages they know. The project is a transatlantic collaborative effort with US researchers partnering with Spanish researchers. High school, undergraduate, graduate, and postdoctoral trainees will receive training in cutting-edge computational and cognitive neuroscience and psycholinguistics, including data science and modeling skills transferable to a range of academic and non-academic STEM careers. One of the fundamental questions that researchers will explore is how two or more languages co-exist within a single language system and how each is represented in the brain. Some prior research suggests there is a deep continuous coactivation of all the languages a person knows even when they are in a single language context, while other research suggests that under many circumstances, only the language relevant in the moment is activated. The project will use the tools of computational neuroscience to develop cognitive theories and implemented models of bilingual and trilingual language processing, which the research team will compare to neuroimaging data with high temporal resolution (magnetoencephalography or MEG). MEG will be collected while monolingual, bilingual, and trilingual individuals process speech from languages they know under conditions designed to promote attention to a single language (isolated words or continuous speech from only 1 language) or two languages (random interleaving of isolated words from 2 languages, or more ecological 'code-switching' between 2 languages). Researchers will use a state-of-the-art neural network model of human speech recognition developed with previous NSF support. They will use continuous speech tracking to relate neural activity to both theoretically-generated hypotheses regarding potential impacts of language co-activation and the behavior and internal activity of neural network models. In this way, researchers will be able to compare human brain responses and neural network model responses to statistical predictions of the expectation level for each successive speech sound (consonant or vowel) and word during presentation of continuous speech. Comparing different models to neural responses will help researchers address fundamental questions, such as whether all the languages a person knows are active whenever they hear any language, and whether this is helpful or causes interference. This research promises to deepen our understanding of multilingual language development and processing in the human brain. A companion project is being funded by the State Research Agency, Spain (AEI). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.