CRII: SHF: Toward Sustainable Software for Science - Implementing and Assessing Systematic Testing Approaches for Scientific Software

Montana State University, Bozeman MT

Investigators

Abstract

Scientific software is widely used in science and engineering. In addition, results obtained from scientific software are used as evidence in research publications. Despite the critical usage of such software, many studies have pointed out a lack of systematic testing of scientific software. As a result, subtle program errors can remain undetected. There are numerous reports of subtle faults in scientific software causing losses of billions of dollars and the withdrawal of scientific publications. This research aims to develop automated techniques for test oracle creation, test case selection, and develop methods for test oracle prioritization targeting scientific software. The intellectual merits of this research are the following: (1) It advances the understanding of the scientific software development process and investigates methods to incorporate systematic software testing activities into scientific software development without interfering with the scientific inquiry, (2) It forges new approaches to develop automated test oracles for programs that produce complex outputs and for programs that produce outputs that are previously unknown, (3) It develops new metrics to measure the effectiveness of partial test oracles and uses them for test oracle prioritization, (4) It extends the boundaries of current test case selection to effectively work with partial or approximate test oracles. The project's broader significance and importance are (1) produces a publicly available, easy to use testing tool that can be incorporated into the scientific software development culture such that the testing activities will not interfere with ?doing science,? (2) recruits Native Americans and women into computer science research, (3) develops a new higher level undergraduate course titled ?Software development methods for Scientists? targeting senior level undergraduate students in non-CS disciplines. This project develops METtester: an automated testing framework that can effectively test scientific software. This testing framework analyzes the source code of the program under test and utilizes machine learning techniques in order to identify suitable test oracles called metamorphic relations (MRs). Then, it automatically generates effective test cases to conduct automated testing based on the identified MRs using a mutation based approach. After that, it creates a prioritized order of MRs to be used with testing in order to identify faults as early as possible during the testing process. Finally, METtester conducts testing on the program under test using the prioritized order of MRs with the generated test cases.