A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites

TL;DR

This paper introduces AgoneTest, an automated system leveraging large language models to generate and evaluate complex, class-level Java test suites, addressing previous limitations of small-scale, manual, and less realistic test evaluations.

Contribution

The paper presents a scalable automated system for generating and assessing class-level Java test suites using LLMs, along with a new dataset and evaluation methodology for real-world applicability.

Findings

Automated system successfully generates complex Java test suites.

Evaluation methodology provides detailed assessment of test quality.

New dataset enables comparison between human-written and generated tests.

Abstract

Unit tests represent the most basic level of testing within the software testing lifecycle and are crucial to ensuring software correctness. Designing and creating unit tests is a costly and labor-intensive process that is ripe for automation. Recently, Large Language Models (LLMs) have been applied to various aspects of software development, including unit test generation. Although several empirical studies evaluating LLMs' capabilities in test code generation exist, they primarily focus on simple scenarios, such as the straightforward generation of unit tests for individual methods. These evaluations often involve independent and small-scale test units, providing a limited view of LLMs' performance in real-world software development scenarios. Moreover, previous studies do not approach the problem at a suitable scale for real-life applications. Generated unit tests are often evaluated via manual integration into the original projects, a process that limits the number of tests executed and reduces overall efficiency. To address these gaps, we have developed an approach for generating and evaluating more real-life complexity test suites. Our approach focuses on class-level test code generation and automates the entire process from test generation to test assessment. In this work, we present AgoneTest: an automated system for generating test suites for Java projects and a comprehensive and principled methodology for evaluating the generated test suites. Starting from a state-of-the-art dataset (i.e., Methods2Test), we built a new dataset for comparing human-written tests with those generated by LLMs. Our key contributions include a scalable automated software system, a new dataset, and a detailed methodology for evaluating test quality.