SoD-HCER: Evaluation of Complex Designs--A Comparative Study

University Of Maryland, College Park, College Park MD

Investigators

Abstract

A critical requirement for any science is the ability to measure. Without measurement, it is difficult to evaluate and improve. Not only is our ability to measure and evaluate software designs today weak, we also lack a thorough understanding of the shortcomings of the strengths and weaknesses of existing software design evaluation methods. To learn more about the shortcomings of current evaluation methods, the PI will in this project study two software products from the systems domain which share the important design criteria of modifiability and performance, the Click modular router and the Jetty http server and servlet container. For each of these two programs, there exist clear design goals and extensive change histories. Using change history data, the PI will create an approximate "master list" of modifiability and performance flaws in early versions of each program. For example, a feature request that required changes to many files could be flagged as a modifiability flaw. She will then apply several evaluation methods (e.g., quantitative software metrics, architecture-based evaluation, and semiautomatic methods such as design snippets) to early versions of each subject program, and compare the output of each method to the approximate master list of design flaws. Project outcomes will include establishment of a benchmark and experimental protocol that can be used to critique new techniques or tools for evaluating software designs using change history logs, and an increased understanding of the deficiencies of existing techniques and tools for evaluating software designs. Broader Impacts: Software designs matter because software matters. To improve software designs, we must have adequate tools for evaluating candidate designs. This project will provide insight into new tools and techniques for software design evaluation. Programmers using these new tools and techniques will produce better software designs and thus better software. In addition, the benchmark produced by this experiment will prove valuable to software engineering educators; instructors will be able to design assignments and projects that involve the benchmark, and which will expose students to issues in evaluating software designs and empirical software engineering research.