CAREER: Uncovering the brain circuitry of language and its interaction with other modalities

Carnegie Mellon University, Pittsburgh PA

Investigators

Abstract

This project aims to better understand how different brain areas interact with each other to enable understanding and production of language in complex real-life settings. Language is a crucial part of human experience, versatile and intimately connected to other abilities. For example, reading a story relies not only on language processing but also on social reasoning, inference of subsequent events, visual imagery, etc. However, due to the complexity of language, most neuroimaging experiments have focused on one aspect of language (e.g., syntax) and studied language in isolation from other cognitive abilities. This has led to theories about which brain areas are involved in language processing and the broad role of each area, but has not been able to reveal what individual units of meaning are processed in each area, how information flows between areas, and how these areas interact with other areas that process other cognitive abilities such as vision or social reasoning. This proposal brings together more powerful experimental data of language in a natural context and uses machine learning, statistics and advances in artificial intelligence to study real-world language use. This project seeks to further understanding of the neurobiology of language and its interaction with other modalities, while also improving AI methods by jointly modeling brain activity and stimulus. Aim 1 will focus on using intracortical recordings (stereoencephalography) to trace the flow of information in the language system at a fine spatial and temporal resolution, enabling more precise discovery of where language features are processed and how information passes between regions. Aim 2 will connect language to visual and social processing, using extensive functional magnetic resonance imaging (fMRI) data from individuals watching a popular sitcom to model the interactions between the language system and other brain systems. Aim 3 will extend this approach to relating brain activity to behavior. A new fMRI dataset where participants engage in natural conversation will be collected and used to study the underlying high-level brain representations. The proposed multidisciplinary research will lead to new educational material as well as new opportunities for students to explore the use of machine learning and AI for science. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.