Collaborative Research: SHF: Small: Automated Quantitative Assessment of Testing Difficulty

University Of California-Santa Barbara, Santa Barbara CA

Investigators

Abstract

Our society is heavily reliant on software systems running on an increasingly large number of programmable devices that surround us. Moreover, the amount of software in safety-critical systems such as cars and planes keeps increasing. Software-quality assurance is one of the most fundamental problems that we are facing in this modern computing-dominated era. One can read about dependability and security problems caused by poor-quality software in the news everyday. It is extremely crucial to develop techniques that can improve the quality of software systems before they cause disastrous consequences during operation. The most common software-quality assurance technique is software testing. Although there has been a surge of progress in automated software-testing techniques, it is hard to predict their effectiveness. Given a piece of software, there is no existing technique that can predict how challenging it will be to automatically test that piece of software. In this project the goal is to develop techniques for assessing the difficulty of automatically testing software. Existing software-complexity metrics do not provide meaningful assessments of testing difficulty. This project's goal is to develop scalable techniques that can provide a quantitative assessment of testing difficulty. In order to be scalable and practical, the method has to rely on a level of abstraction that provides efficient analysis, while preserving meaningful characteristics of program behavior that relate to testing difficulty. The approach used in this project builds on two concepts that provide a promising abstraction for quantitative assessment of testing difficulty: 1) path complexity, and 2) path selectivity. Path complexity assesses how the number of paths in a given program increases with increasing execution depth, and path selectivity assesses the difficulty of finding values that satisfy a path condition. The team of researchers working on this project will develop techniques that automatically compute path complexity and path selectivity and then combine them to obtain a quantitative measure for testing difficulty. By developing techniques that can assess software-testing difficulty, this project will enable development of more effective software-testing techniques based on better resource allocation for software-quality assurance tasks. This will lead to improvements in software quality, and reduction in software defects that cause dependability and security problems. Secondly, the research activity will help to expose graduate and undergraduate students to software-quality assurance challenges and techniques. Finally, the research activity will help to disseminate the knowledge, techniques and tools developed within the scope of this project through publishing in open literature and making available the software tools that are developed as open source. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.