ASTER: Natural and Multi-language Unit Test Generation with LLMs

TL;DR

This paper explores how large language models, guided by static analysis, can generate high-quality, natural, and multi-language unit tests, demonstrating competitive coverage and developer-friendly readability.

Contribution

It introduces a generic pipeline leveraging LLMs and static analysis for multi-language unit test generation, improving coverage and naturalness over existing methods.

Findings

LLM-guided test generation achieves high code coverage.

Generated tests are more natural and understandable to developers.

The approach is effective across Java and Python, including complex scenarios.

Abstract

Implementing automated unit tests is an important but time-consuming activity in software development. To assist developers in this task, many techniques for automating unit test generation have been developed. However, despite this effort, usable tools exist for very few programming languages. Moreover, studies have found that automatically generated tests suffer poor readability and do not resemble developer-written tests. In this work, we present a rigorous investigation of how large language models (LLMs) can help bridge the gap. We describe a generic pipeline that incorporates static analysis to guide LLMs in generating compilable and high-coverage test cases. We illustrate how the pipeline can be applied to different programming languages, specifically Java and Python, and to complex software requiring environment mocking. We conducted an empirical study to assess the quality of the generated tests in terms of code coverage and test naturalness -- evaluating them on standard as well as enterprise Java applications and a large Python benchmark. Our results demonstrate that LLM-based test generation, when guided by static analysis, can be competitive with, and even outperform, state-of-the-art test-generation techniques in coverage achieved while also producing considerably more natural test cases that developers find easy to understand. We also present the results of a user study, conducted with 161 professional developers, that highlights the naturalness characteristics of the tests generated by our approach.