ITR/SY: A Neuromorphic Vision System for Every-Citizen Interfaces

California Institute Of Technology, Pasadena CA

Investigators

Abstract

This project aims to extend an existing simple saliency-based visual attention system to animated color video sequences so as to enable it to cue the object recognition module towards interesting locations in live video streams, and simultaneously to extend an existing model for object recognition to on-line adaptability through top-down signals and task- and object-dependent learning of features. The PIs will then integrate these attention and recognition models, by developing feedforward and feedback interactions between localization of regions of interest and object recognition in those regions. This will require substantial elaboration of both models, as well as specific work on their integration. The result will be a complete model of object localization and recognition in primates, with direct applicability to computer vision challenges. The PIs will next implement and deploy the combined model on a cluster of CPUs linked by very fast interconnect (just installed at USC) to allow for real-time processing, and will demonstrate its utility in a prototype video-conferencing application in which the on-line adaptive attentional component of the integrated system will quickly locate regions in the monitored environment where something interesting is happening (e.g., a user raising her hand in a conference room). The recognition part of the system will then be trained and refined on-line to recognize relatively simple hand signs (e.g., a finger pointing up, meaning that the user wishes to become the center of interest in a video-conference). This work will demonstrate two points: that a biologically-inspired approach to traditionally hard computer vision problems can yield unusually robust and versatile vision systems (which work with color video streams and quickly adapt to various environmental conditions, users, and tasks); and that computational neuroscience models of vision can be extended to yield real, useful and widely applicable computer vision systems, and are not restricted to testing neuroscience hypotheses under simple laboratory stimuli.