Collaborative Research: Neural and computational models of spatio-temporally varying natural scenes

Brown University, Providence RI

Investigators

Abstract

As we move through our visual environment, the pattern of light that enters our eyes is strongly shaped by the properties of objects within the environment, their motion relative to each other, and our own motion relative to the external world. This collaborative project will quantify motion within natural scenes, record activity from populations of neurons in the early visual pathway in response to the motion, and develop models of motion representation across neuronal populations. The primary goals of the work are to fully characterize the biological representation of motion in natural scenes in the early stages of visual processing that sets the stage for cortical computation critical for visual perception, and to unify the biological findings with computational models of motion from the computer vision community. The perception of visual motion is critical for both biological and computer vision systems. Motion reveals structure of the world including the relative and absolute depths of objects, surface boundaries between objects and information about ego-motion and the independent motion of other objects. The effects of visual motion on the relationship between spatially localized and global properties of the natural visual scene, and how this is represented by the early visual pathway of the brain, are largely unknown. This project addresses the computation of local and global properties of natural visual scenes by both distributed neural systems and computer vision algorithms using a novel set of complex naturalistic stimuli in which ground truth properties of the scene are known, and all aspects of the scene, including its reflectance, surface properties, lighting and motion are under investigator control. A unified probabilistic modeling framework will be adopted, that ties together the computational and biological models of properties of the natural scene. Neural activity will be recorded from a large population of densely sampled single neurons from the visual thalamus. From the perspective of the computer vision community, an important challenge exists in inferring the motion of the external environment (or "optical flow") from sequences of 2D images. From the perspective of the neuroscience community, quantifying the distributed neural representation of luminance and motion in the early visual pathway will be a critical step in understanding how scene information is extracted and prepared for processing in higher visual centers. A team of investigators with experience in computer science, engineering, and neuroscience will develop a theoretical foundation and rich set of methods for the representation and recovery of local luminance, local motion boundaries and global motion by brains and machines.