EAGER: Top-down processes to extract meaning from images

Children'S Hospital Corporation, Boston MA

Investigators

Abstract

Humans rapidly process images and scenes so they can understand what is occurring in the world around them. Humans do this so well that they outperform existing computational models' ability to understand what elements exist in the scene, their location, or the actions they are involved in. One limitation of computational models is that they do not provide a detailed interpretation of a scene's individual components the way that humans do. For example, the computational model may successfully label an image as containing a horse, but humans will also naturally identify smaller components of the horse, such as the eyes, ears, mouth, mane, legs, tail, and so on. Identifying these individual components and their relationships is an essential part of human visual processing. Such differences in visual understanding create a challenge for constructing artificial computational systems that see and interpret the world similarly to humans. These fundamental limitations are related to the fact that existing computational systems rely primarily on what is called 'bottom-up processing', the sequential processing of visual features from simple to complex ones, which does not account for how human cognition influences meaningful recognition of the image. Our main goal is to investigate the computational principles and neurobiological systems that allow for integration of cognitive experience within visual processing. We combine psychological studies in humans, neurophysiological recordings of brain tissue, and computational work to build an integrative model capable of extracting complex meaning from images, in a way that more closely resembles human capabilities. The research will have broad implications in understanding how the brain processes images and the neural circuits that are involved. Additionally, the insights obtained from this project could have applications in a broad range of domains including robot vision, automatic navigation, surveillance, and automatic clinical image understanding. As part of the project we will establish a summer course based on the research products in which we will train the next generation of scholars at the interface of brains, minds, and machines. In the human brain, information flows both from low to higher visual areas, as well as in the opposite direction throughout ventral visual cortex. This bi-directional processing has a fundamental role in cortical computations. Yet, the functions implemented by the top-down components form an open problem in visual cognition. Understanding the limitations of feed-forward processing (from lower to higher visual regions) will shed light on the mechanisms by which prior knowledge is integrated with bottom-up inputs, and guide development of new algorithms for extracting useful meaning from sensory input. Applications of deep network models now play a significant role in machine learning across multiple domains, but these fail to capture fundamental aspects of visual processing. Our research program examines the possibility that existing feed-forward recognition models constitute a first stage leading to the initial activation of category candidates, which is incomplete and often inaccurate. The first stage then triggers the application of class-specific processes, which recover a richer and more accurate interpretation of the visible scene, and reject initial false candidates. The proposal involves three main components: (i) Psychophysics experiments to evaluate the accuracy and speed of how humans extract meaning from images; (ii) Invasive neurophysiological recordings along the human ventral visual cortex to understand the neural circuits involved in extracting meaning from a novel data set of minimal images; (iii) A computational model that integrates bottom-up computations with top-down signals to extract meaning from images. The research efforts will be combined with educational and outreach activities aimed at disseminating the scientific insights and incorporating cutting-edge research into training opportunities for undergraduate and graduate students.