CSR--EHS: Robust Testing by Testing Robustness of Embedded Systems

University Of Pennsylvania, Philadelphia PA

Investigators

Abstract

As modern embedded systems gain more functionality and complexity, there is a need for a novel discipline for their design, development and deployment. In recent years, the idea of the model-based design paradigm is to develop design models and subject them to early analysis, testing, and validation prior to their implementation. Simulation-based testing ensures that a finite number of user-defined system trajectories meet the desired specification. Even though computationally inexpensive simulation is ubiquitous in system design, it suffers from incompleteness, as it is impossible or impractical to test all system trajectories. On the other hand, verification methods enjoy completeness by showing that all system trajectories satisfy the desired property. For embedded hybrid systems with an infinite number of possible behaviors, exhaustive verification seems to be very hard, and simulation-based testing seems to provide no confidence in our system design. In addition to the gap between testing and verification for embedded systems, there is even a more fundamental, and largely unaddressed, challenge. Uncertainty in the environment, errors in physical devices make overall system robustness one of most critical yet least understood challenges in embedded systems. There is a clear intellectual opportunity for laying the scientific foundations and developing methods and algorithms for analyzing and testing the robustness and safety of embedded hybrid systems. This project brings together leading experts in embedded control, hybrid systems, and software monitoring and testing to develop the foundations of a modern framework for testing the robustness of embedded hybrid systems. The central idea that this proposal is centered around is the notion of a robust test, where the robustness of nominal test can be computed and used to infer that a tube of trajectories around the nominal test will yield the same qualitative behavior. By computing the robustness margins of tests, this project explores how to infer how robust each test is, guide subsequent tests, estimate the robustness for the system, as well provide well-defined coverage metrics using finite number of tests. In addition, this project emphasizes cross-cutting, multi-departmental education of graduate students and emphasizes testing and robustness for embedded hybrid systems in relevant electrical engineering and computer science courses. The educational agenda is to expose computer science students to notions of robustness, and control students to software testing algorithms.