US-German Data Sharing Proposal: CRCNS Data Sharing: REvealing SPONtaneous Speech Processes in Electrocorticography (RESPONSE)

Virginia Commonwealth University, Richmond VA

Investigators

Abstract

The uniquely human capability to produce speech enables swift communication of abstract and substantive information. Currently, nearly two million people in the United States, and far more worldwide, suffer from significant speech production deficits as a result of severe neuromuscular impairments due to injury or disease. In extreme cases, individuals may be unable to speak at all. These individuals would greatly benefit from a device that could alleviate speech deficits and enable them to communicate more naturally and effectively. This project will explore aspects of decoding a user's intended speech directly from the electrical activity of the brain and converting it to synthesized speech that could be played through a loudspeaker in real-time to emulate natural speaking from thought. In particular, this project will uniquely focus on decoding continuous, spontaneous speech processes to achieve more natural and practical communication device for the severely disabled. The complex dynamics of brain activity and the fundamental processing units of continuous speech production and perception are largely unknown, and such dynamics make it challenging to investigate these speech processes with traditional neuroimaging techniques. Electrocorticography (ECoG) measures electrical activity directly from the brain surface and covers an area large enough to provide insights about widespread networks for speech production and understanding, while simultaneously providing localized information for decoding nuanced aspects of the underlying speech processes. Thus, ECoG is instrumental and unparalleled for investigating the detailed spatiotemporal dynamics of speech. The research team's prior work has shown for the first time the detailed spatiotemporal progression of brain activity during prompted continuous speech, and that the team's Brain-to-text system can model phonemes and decode words. However, in pursuit of the ultimate objective of developing a natural speech neuroprosthetic for the severely disabled, research must move beyond studying prompted and isolated aspects of speech. This project will extend the research team's prior experiments to investigate the neural processes of spontaneous and imagined speech production. In conjunction with in-depth analysis of the recorded neural signals, the researchers will apply customized ECoG-based automatic speech recognition (ASR) techniques to facilitate the analysis of the large amount of phones occurring in continuous speech. Ultimately, the project aims to define fundamental units of continuous speech production and understanding, illustrate functional differences between these units, and demonstrate that representations of spontaneous speech can be synthesized directly from the neural recordings. A companion project is being funded by the Federal Ministry of Education and Research, Germany (BMBF)