CAREER: Reinventing Computer Vision through Bio-inspired Retinomorphic Vision Sensors, Corticomorphic Compute-In-Memory Processors and Event-based Algorithms

University Of Pittsburgh, Pittsburgh PA

Investigators

Abstract

State-of-the-art computer vision (CV) pipelines are compute/memory intensive and power hungry making them unsuitable for high-speed applications such as hypersonic missile tracking or resource-deficit edge applications such as autonomous drone navigation due to size, weight and power (SWaP) constraints. Neuromorphic engineering is a promising frontier to usher in the next generation of CV systems taking advantage of sparsity in the input and network architecture, reducing the number of operations through event-based computation i.e., compute only when necessary. This project aims to develop a versatile energy-efficient bio-inspired sensing, computing, and learning framework by developing a closely-knit system, from devices and circuits with rich spatio-temporal dynamics to network architectures inspired by the visual cortex and adaptive learning algorithms for visual perception. This will be achieved primarily using compute-in-memory (CIM) architectures that process and extract a variety of critical visual features in close physical proximity to where the data is stored in memory. The proposed research will embark on a uniquely integrated approach that addresses challenges at all levels, from devices, circuits, architectures, and algorithms leading to novel CV applications, inspired by neuroscience, such as low latency dynamic object classification, tracking and adaptive visual attention. The breadth of skillsets that are required to effectively train a new cadre of workforce in neuromorphic engineering for computer vision makes curriculum design and integration with existing frameworks incredibly challenging. The proposed BioVision educational consortium will address this issue. The main objective of this consortium is to collaborate and implement a comprehensive workforce development plan that incorporates evidence-based best practices to help train a new generation of engineers and researchers, who are equipped to satisfy the growing needs of the computer vision industry. The grand vision of this proposal is to reimagine modern computer vision (CV) pipelines that exist today and replace the components with bio-inspired sensors, processors and algorithms that can drastically improve energy efficiency, data efficiency and lower latency. To reinvent the CV pipeline, three research thrusts will be addressed simultaneously. Thrust 1 will focus on creating and building a new class of retina-inspired vision sensors, that outperforms existing cameras, such as frame-based or neuromorphic Dynamic Vision Sensors (DVS), in terms of features, efficiency and latency. Thrust 2 will focus on modeling, design and implementation of scalable corticomorphic networks on hardware, exhibiting non-linear neuromodulatory dynamics at multiple timescales using mixed-feedback control. Thrust 3 will focus on implementation of network architectures and algorithms inspired by neuroscience, such as reinforcement learning with stochastic rewards, event-based temporal pattern recognition. The proposed research has the potential to lead a generational shift in the fields of computer vision, neuromorphic computing, and artificial intelligence. Developing an energy-efficient event-based camera capable of versatile spatiotemporal pattern recognition and novel features inspired by the retina, along with a general purpose, programmable, event-based computer vision pipeline can have a transformative impact on our society, by impacting critical areas like healthcare, Internet of Things (IoT), military defense, edge computing and industrial automation. Enabling the use of advanced CV on personal electronics can revolutionize our lifestyle through technologies such as self-driving vehicles, always-on smart surveillance, and virtual/augmented reality (VR/AR) applications. Bio-inspired vision sensors, such as the DVS camera sold by companies like Prophesee and iniVation, are primarily developed in Europe and Asia and have no industry or academic contribution from USA. This proposal will address this national challenge by training a new generation of world-class researchers and provide the USA with a leading advantage in the deployment of next-generation computer vision systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.