ChatUniTest: A Framework for LLM-Based Test Generation

TL;DR

ChatUniTest is an innovative LLM-based framework for automated unit test generation that improves accuracy and coverage through adaptive context and validation mechanisms, outperforming existing tools in several projects.

Contribution

The paper introduces ChatUniTest, a novel framework with adaptive context and validation-repair mechanisms, enhancing LLM-based unit test generation effectiveness.

Findings

Outperforms TestSpark and EvoSuite in half of the evaluated projects.

Achieves the highest overall line coverage among compared tools.

User study confirms substantial value for stakeholders.

Abstract

Unit testing is an essential yet frequently arduous task. Various automated unit test generation tools have been introduced to mitigate this challenge. Notably, methods based on large language models (LLMs) have garnered considerable attention and exhibited promising results in recent years. Nevertheless, LLM-based tools encounter limitations in generating accurate unit tests. This paper presents ChatUniTest, an LLM-based automated unit test generation framework. ChatUniTest incorporates an adaptive focal context mechanism to encompass valuable context in prompts and adheres to a generation-validation-repair mechanism to rectify errors in generated unit tests. Subsequently, we have developed ChatUniTest Core, a common library that implements core workflow, complemented by the ChatUniTest Toolchain, a suite of seamlessly integrated tools enhancing the capabilities of ChatUniTest. Our effectiveness evaluation reveals that ChatUniTest outperforms TestSpark and EvoSuite in half of the evaluated projects, achieving the highest overall line coverage. Furthermore, insights from our user study affirm that ChatUniTest delivers substantial value to various stakeholders in the software testing domain. ChatUniTest is available at https://github.com/ZJU-ACES-ISE/ChatUniTest, and the demo video is available at https://www.youtube.com/watch?v=GmfxQUqm2ZQ.