CI-EN: Collaborative Research: TraceLab Community Infrastructure for Replication, Collaboration, and Innovation

University Of Notre Dame, Notre Dame IN

Investigators

Abstract

This is the second phase of the TraceLab project, which was initially funded under an NSF Major Research Instrumentation grant. The goal of this project is to deliver an instrument that facilitates reproducibility of Software Engineering experiments, fosters comparative evaluation, and provides an environment in which components can be easily shared across research groups. The challenge of experimental reproducibility extends across almost every science and engineering discipline. Recently, a widely reported study conducted by the biotech firm "Amgen" revealed that of 53 previously published landmark papers, only six were reproducible. Recent studies have unearthed similar problems across a diverse set of Software Engineering domains, including, but not limited to, software traceability, feature location, and compiler optimization. Reproducibility is often undermined by lack of publicly available datasets, obsolete and unavailable tools, insufficient details about the experiment, and undocumented decisions about the way various metrics are computed. TraceLab provides a plug-and-play experimental environment, as well as libraries of shareable components and seminal experiments. As such, it is designed to address these problems. TraceLab introduces a radically different way of approaching empirical software engineering research and paves the way for greater community collaboration, more rigorous evaluation of results, and an easier entry-way for new researchers. It is expected to lay the foundation for future advances in the field of empirical software engineering, accelerate and shape future research directions, support industrial pilot studies, and significantly reduce the cost and effort that oftentimes discourages new researchers from entering the field. In addition, the project will provide opportunities to a diverse group of undergraduate and graduate students, and will be used for educational purposes across various software engineering courses.