Collaborative Research: SLES: Bridging offline design and online adaptation in safe learning-enabled systems

University Of Pennsylvania, Philadelphia PA

Investigators

Abstract

Maintaining the safety of a learning-enabled system that navigates in an unknown environment is a major challenge owing to uncertainty in the environment, the system's goals, and the system's learning-enabled components. This project proposes a novel approach to mitigating these uncertainties. The project’s novelties are the development of a two-phase design and deployment process integrated into a tight feedback loop: (1) an offline design process aimed at synthesizing systems that are provably robust and resilient to known unknowns, and (2) an automated online safety monitoring phase, during which a deployed learning-enabled system seeks to detect, learn about, and adapt to unknown unknowns. By closing the loop between online safety monitoring and offline design, using data collection as the enabling modality connecting these two phases, meaningful notions of both offline and online safety can be defined. The project’s impacts include: (i) a mathematical guarantee on the end-to-end safety of the design and deployment process described above for learning-enabled systems; (ii) methods that ensure safety with respect to known unknowns during the offline design stage, and safety with respect to unknown unknowns during deployment, when possible; and (iii) techniques that identify and learn about unknown unknowns, that is, novel sources of uncertainty, so that they can be integrated into the design of future systems. Realizing the project’s technical objectives requires major advances in representing, characterizing, and accounting for uncertainty in learning-enabled components using streaming data generated from dynamic distributions. The project addresses these challenges by first developing novel safety-rich data augmentation and domain randomization techniques for the training of safe learning-enabled systems. The project also seeks to identify the correct types of safety-rich data to be collected to ensure end-to-end safety of a system with learning-enabled components trained using this data. These data generation and augmentation techniques are integrated into novel safety-aware robust learning, control, and verification methods with strong safety guarantees. Finally, the project aims to develop online safety monitoring, uncertainty quantification, and adaptation techniques for contending with unknown unknowns during deployment. Meeting these objectives requires novel techniques rooted in conformal prediction and active learning that allow for principled tradeoffs between risks to system safety and active data collection and learning, thus closing the design and deployment loop. The project outcomes are incorporated into undergraduate and graduate classes at both Penn and UC Berkeley, and the research team plans to organize workshops at major controls, machine learning, and cyber-physical systems conferences to help foster a novel interdisciplinary community of safe learning-enabled systems researchers. All members of the research team are committed to promoting diversity and inclusion within their research groups. This research is supported by a partnership between the National Science Foundation and Open Philanthropy. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.