SHF: Medium: Collaborative Research: Enhancing Continuous Integration Testing for the Open-Source Ecosystem

University Of Illinois At Urbana-Champaign, Urbana IL

Investigators

Abstract

Continuous integration (CI) is an important software development activity that aims to improve software development by automating software compilation and regression testing. Recent studies report that CI helps developers deploy faster and reduce development cost. Given these success stories, CI has attracted rapidly increasing interest and adoption, e.g., Travis CI, the currently most popular CI service, is used by over 300,000 GitHub projects. Despite the success of CI, developers report they would like to see improvements in CI. First, they want to faster obtain regression test results. Second, they want better handling of so-called flaky tests, which are regression tests that can non-deterministically pass or fail, and whose failures negatively affect developer's productivity. Third, developers report that CI builds do not provide sufficient debugging assistance for reasoning about failed regression tests. While regression testing has been studied for over three decades, it has not been studied in the context of CI until recently. To substantially improve regression testing in CI, the PIs propose to develop novel techniques and tools that address three important challenges: (1) test selection to speed up regression testing and the development cycle, (2) test reliability to mitigate the problems that flaky tests introduce, and (3) debugging assistance to ease the effort of diagnosing and fixing the true and flaky regression test failures. The PIs plan to develop techniques and tools based on a mix of static and dynamic program analyses, leveraging not only information from two project revisions (as traditional in regression testing) but also from all historical build and testing information available in CI testing. The PIs plan to embody their techniques in a tool-set and evaluate them extensively on open-source projects and in industrial collaborations. The broader impacts of enhancing continuous integration testing are to allow software developers to faster build higher quality software, which can benefit our modern society that greatly depends on software. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.