Text, Neuroimaging, and Memory: Unified Models of Corpora and Cognition

Princeton University, Princeton NJ

Investigators

Abstract

The PIs will develop new machine learning algorithms to explore how meaning is represented in the brain and how meaning representations shape human memory. Current neuroscientific theories of memory posit that forming a memory for a particular event involves associating the details of that event with the person's current mental context, i.e., everything else that she is thinking about at the time. When trying to remember the event, the person can access stored details by reinstating the mental context that was present when the memory was formed. This fits with the intuition that forgotten details (e.g., the location of misplaced house keys) can be retrieved by mentally "re-tracing steps", i.e., trying to reinstate the mindset that was present at the time of the original event. With these theories in mind, the goal of this work is to develop machine learning algorithms that make it possible to track, based on fMRI brain data and behavioral memory data, the process of "mentally re-tracing steps"---the proposed algorithms will be able to decode the state of a person's mental context as she forms memories and (later) as she searches for these memories. The proposed work uses two fundamental ideas about memory and meaning: The first idea is that mental context is shaped by the meanings of recently encountered stimuli. The second idea is that semantic relationships between concepts in the brain mirror statistical relationships between words in naturally occurring language. The developed algorithms will bring together data from three sources---behavioral data from subjects performing memory recall tasks, fMRI neuroimaging data collected while subjects performed these tasks, and large collections of documents---to discover a latent meaning space that can simultaneously describe all three types of information. Each point in this space describes a mental context. Thus the core of the proposed work is to develop latent variable models and algorithms that can infer from data how the mental context moves through meaning space as a person stores and searches for memories. The proposed work will lead to fundamental advances in machine learning (new algorithms for inferring hidden variables based on multiple, heterogeneous data types) and neuroscience (more refined theories of how memory search is accomplished in the brain). Furthermore, this work will catalyze the development of new technologies for diagnosing and remediating memory problems, by making it possible to track how the contextual reinstatement process is going awry in people experiencing memory retrieval failure.