SHF: Small: Automated Unit Test Generation using Large Language Models

Northeastern University, Boston MA

Investigators

Abstract

Large Language Models (LLMs) are systems that rely on machine learning techniques to solve queries that are stated using natural language. To interact with an LLM, a user provides a textual description, commonly referred to as a "prompt", of the problem to be solved. In response to receiving a prompt, an LLM generates one or more textual results that represent solutions to the problem. LLMs have recently become very popular for applications such as translation and chatbots. Furthermore, LLMs are starting to be adopted for applications in software development such as automatic code completion in Integrated Development Environments used by programmers. This project explores the use of LLMs for automatically testing software. To this end, tests are generated by providing LLMs with prompts containing code fragments of the software under test and artifacts associated with the software such as documentation and usage examples. The project will result in improved testing of software, thereby improving its quality. Moreover, the project will unburden developers from the chore of writing tests manually, thereby increasing their productivity. The developed automatic test generation techniques based on Large Language Models (LLMs) will feature a feedback-directed, iterative approach. In each iteration, an LLM is given a prompt containing the signature of a function under test, along with usage examples, test framework code, and results obtained from previously generated tests. The resulting completions consist of candidate tests, which are checked for syntactic validity and executed to check if they pass or fail. Analysis of the execution behavior of failing tests will direct the construction of refined prompts. The project will target both dynamically typed and statically typed programming languages. The work will be evaluated by comparing the generated test suites against those produced by state-of-the-art test generation tools. All developed test generation tools will be made available as open-source software for others to use and build on, and results will be disseminated via publications. Societal benefits will follow from improvements in software quality enabled by the adoption of the developed techniques. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.