CAREER: CompCog: Reverse-engineering neural mechanisms of object cognition with multilevel computational modeling

Yale University, New Haven CT

Investigators

Abstract

Imagine pointing a camera at a child building a tower with blocks to take a picture of that scene. In a split second, both the camera and your retina will register the light bouncing off of the surfaces within the view, but there is a great difference between these raw sensory measurements of light and what you experience as perceptions and thoughts --- what the child’s face might look like in a different viewing angle, the shape and posture of the body, the configuration of the tower they are building and whether it’s a stable tower, and which piece they are about to place next. This project studies algorithms by which the light that arrives at our retinas is transformed into these perceptions and thoughts, including objects’ 3D shapes and physical properties, predictions and mental simulations of what will or could happen next in the scene, and our plans to realize those futures. This project examines these “object cognition” abilities by creating new computational models of how we see and think and compares the internal structure and performance of these models to high-resolution brain activity and task performance in both humans and non-human primates. The overarching hypothesis of this project is that the brain implements object cognition by building and manipulating generative models of how physical scenes form, dynamically unfold, and project onto sensory inputs. To test this hypothesis, the research team aims to build novel multilevel computational theories and models that natively interoperate at both the cognitive level of structure-preserving object representations and the neural level of distributed, multi-area codes. Across three studies, the team tests a range of alternatives implementing competing hypotheses, with psychophysics and high-resolution electrophysiological data from human and non-human primate brains, curating a new large-scale dataset of intracranial EEG recordings of the brain. These studies aim to reveal neurocomputational principles of object cognition, yielding critical new insights into how the brain implements the mind. These computational advances in human cognition may also inform the development of fundamentally new capabilities in AI, including systems that perceive their environments in much richer and more sophisticated ways, and that plan adaptively and flexibly, and interact safely and in ways better aligned to humans. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.