I-Corps: Control for Visual Scene Perception

Cornell University, Ithaca NY

Investigators

Abstract

The broader impact/commercial potential of the proposed I-Corps project consists of significantly increased safety and control for real-world applications of computer vision and mobile robotics. The first application in the optimal monitoring and control of smart environments will impact the safety and productivity of many industries, such as video surveillance, access control, and smart buildings/cities. Computer-vision software applications will impact industries such as autonomous automobiles by predicting actions of nearby pedestrians, animals, or other vehicles. Automated video processing and recognition software will potentially reduce the impact of catastrophes such as terrorist attacks or mass shootings via intent prediction and anomaly detection available through the proposed video analytics software. Additionally, in a different application the proposed solution may be used to optimize a building's energy efficiency and reduce building operating costs. This I-Corps project further develops a platform for autonomous visual scene perception and feedback control. Autonomous real-time perception and predictive control will dramatically increase the safety and efficiency across several potential applications by directing a human operator's attention to a situation or automatically applying changes to the sensor field of view or controllable environmental conditions such as lighting or temperature. The unique deep learning Bayesian optimization framework does not require prior knowledge of the scene in which it is deployed and instead learns the scene representation over time. The project ties together machine learning, estimation, and control systems. Proof of concept testing has been successfully completed in real-time vehicles equipped with cameras and using deep learning in the loop for autonomous control of the sensor field of view. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.