SHF: MEDIUM: Achieving Software Reliability without True Test Oracles

Columbia University, New York NY

Investigators

Abstract

Conventional software testing checks whether each output is correct for the set of test inputs. But for some software, it is not known what the correct output should be for some inputs -- yet it is still important to detect coding errors in that software, so they can be fixed. This dilemma arises frequently for machine learning, simulation and optimization applications, often "Programs which were written in order to determine the answer in the first place. There would be no need to write such programs, if the correct answer were known." As these kinds of applications are frequently used in public infrastructure and biomedical research (domains targeted in this research), it is critical to detect and fix errors before a calamity occurs. Fortunately, many such applications reflect 'metamorphic properties' that define a relationship between pairs of inputs and outputs, such that for any previous input i with its already known output o, one can easily derive a test input i' and predict the expected output o'. If the actual output o'' is different from o', then there must be an error in the code. This project investigates methodology for determining the metamorphic properties of software and for devising good test cases from which the secondary tests can be derived. The project extends the inputs/outputs considered in previous work on metamorphic testing to focus on application state, before and after, rather than just functional parameters and results. The research also extends the pairwise relations implied by metamorphic properties to 'semantic similarity' for nondeterministic applications, applied to profiles from numerous executions, since an exact relation cannot be expected to hold for a single pair of test executions. These extensions enable treatment of more sophisticated properties that preliminary experiments have shown to reveal defects that were not detected otherwise.