Collaborative Research: RI: Small: A Scalable Architecture for Image Interpretation

New Mexico Consortium, Los Alamos NM

Investigators

Abstract

Seamless understanding of the meaning of visual images is a key property of human cognition that is far beyond the abilities of current computer vision programs. The purpose of this project is to build a computational system that captures the dynamical and interactive aspects of human vision by integrating higher-level concepts with lower-level visual perception. If successful, this system will be able to interpret visual scenes in a way that scales well with the complexity of the scene. Current computer vision systems typically rely on relatively low-level visual information (e.g., color, texture, shape) to classify objects or determine the overall category of a scene. Such categorization is typically done in a "bottom-up" fashion, in which the vision system extracts lower-level features from all parts of the scene, and subsequently analyzes the extracted features to determine which parts of the scene contain objects of interest and how those objects should be categorized. Such systems lack the abilities to scale to large numbers of visual categories and to identify more complex visual concepts that involve spatial and abstract relationships among object categories. Visual perception by humans is known to be a temporal process with feedback, in which lower-level visual features serve to activate higher-level concepts (or knowledge). These active concepts, in turn, guide the perception of and attention given to lower-level visual features. Moreover, activated concepts can spread activation to semantically related concepts (e.g., "wheels" might activate "car" or "bicycle"; "bicycle" might activate "road" or "rider"). In this way there is a continual interaction between the lower and higher levels of vision, which allows the viewer to focus on and connect important aspects of a complex scene in order to perceive its meaning, without having to pay equal attention to every detail of the scene. The system proposed here will model these aspects of human visual perception. The proposed system, called Petacat, will integrate and build on two existing projects: the HMAX model of object recognition originally developed by Riesenhuber and Poggio, and the Copycat model of high-level perception and analogy-making, developed by Hofstadter and Mitchell. HMAX models the "what" pathway of mammalian visual cortex via a feed-forward network that extracts increasingly complex textural and shape features from an image. (HMAX has been reimplemented, as the "Petascale Artificial Neural Network" or PANN, by the Synthetic Vision Group at Los Alamos to allow for high-performance computing on large numbers of neurons.) Copycat implements a process of interaction between high-level concepts and lower-level perception, and has been used to model focus of attention, conceptual slippage, and analogy-making in several non-visual domains. This project will marry the feature extraction abilities of HMAX/PANN with the higher-level interactive perceptual abilities of Copycat to build the Petacat architecture. The image interpretation abilities of Petacat will be evaluated on families of related semantic visual recognition tasks (e.g., recognizing, in a flexible, human-like way, instances of "walking a dog"). The evaluation part of the project will involve the creation of image databases for benchmarking semantic image-understanding systems. The Petacat source code and benchmarking databases will be made publically available via the web.