CRII: FET: Neuromorphic Processing Framework for Spatiotemporal Fusion of Visual Sensors

Kennesaw State University Research And Service Foundation, Kennesaw GA

Investigators

Abstract

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Visual processing tasks such as detection, tracking, and localization are essential to the automation of unmanned aerial vehicles (UAV), robots, surveillance, and defense systems. However, these intelligent tasks become challenging on high-speed motion and edge devices due to limited computing resources and low power supplies. This research will explore a brain-inspired framework to process the visual information from two complementary visual sensors, event-based dynamical vision sensors (DVS) and frame-based standard cameras, in a sensor-fusion style. The overarching goal is to address the challenge of high-speed and energy-efficient visual processing with end-to-end closed-loop control on edge computing systems. The proposed research will benefit numerous robotics, surveillance, IoT security, and national defense applications. This work will also explore novel hybrid neural networks, thus contributing to the quest to general AI and enhancing the interdisciplinary collaboration between computer science and neuroscience. To encourage young students in this research, the project will 1) design hands-on projects and course modules related to DVS cameras and neuromorphic algorithms, 2) initiate K-12 education outreach for local minority high school students through ongoing University Programs, and 3) recruit minority undergraduate researchers. The proposed project will exploit the synergy of two brain-inspired learning models, neuromorphic spiking neural networks and regular deep neural networks. Such a hybrid neuromorphic framework can harness the high spatial resolution from a standard camera and the high temporal resolution from a DVS camera. The temporal encoded data from the DVS camera is suitable to be processed in a spiking neural network. In contrast, the data from the standard camera are compatible with traditional convolutional networks. This proposal will 1) design a hybrid neuromorphic framework composed of spiking neural networks and conventional artificial neural networks to process event-frame fused visual data; 2) adapt such a framework to UAV or robots and develop an end-to-end close-loop neuromorphic platform for various high-speed visual tasks; and 3) explore the model compression of hybrid neural networks and architecture design of the hardware accelerator for the proposed framework. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.