EPCN: VISUALS: Verifiable Information-Theoretic Safety Under Augmented Latent Shifts

University Of Florida, Gainesville FL

Investigators

Abstract

Deep learning-based perception and control are increasingly popular in recent autonomous systems. Unfortunately, deep learning is vulnerable to various changes in the visual environment, known as visual shifts, which threaten the system’s performance and safety. This project aims to ensure the safety and performance of vision-based autonomous systems subject to visual shifts, including sun glares and seasonal changes such as snow-covered terrain. We will build specialized modeling, training, and adaptation techniques to overcome visual shifts based on the core idea that visual uncertainty should be treated by balancing informativeness with conservatism. This balance can be achieved by focusing on the most surprising visual phenomena to make safe choices in unforeseen circumstances. The intellectual merit of this project includes developing theories and algorithms at the intersection of formal verification and information theory that will endow vision-based autonomous systems with high-performance behaviors and previously unavailable theoretical guarantees. The broader impacts of this project include making vision-based autonomy safer and more reliable, particularly in the automotive sector, as well as transitioning insights gained from the project to practice by collaborating with researchers in the auto industry, publicly releasing our data and code, and providing university students with hands-on experience with safe perception and control. This project will develop an end-to-end methodology that leverages information-theoretic and statistical techniques in modeling, analysis, training, control, and adaptation. At the foundation of the proposed methodology is a robust framework for probabilistic verification and control synthesis, which will provide conservative models and safety estimates under latent shifts. Building on these models is a neuro-symbolic training process that bridges the gap between visual perception, safety, and control. Finally, to protect the system from overreacting to unseen shifts, this project will develop an online adaptation framework based on quick change detection and perception/control switching. The proposed methodology will be validated on a variety of autonomous systems from different domains, including physical experiments of small-scale autonomous racing. This research is expected to provide tight complementary connections between information-theoretic learning and formal techniques for safe control. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.