Generating Project-Specific Test Cases with Requirement Validation Intention

TL;DR

This paper introduces IntentionTest, a method that generates project-specific test cases aligned with validation intentions using retrieval and editing with an LLM, improving relevance and success rates.

Contribution

It presents a novel approach combining retrieval and editing with LLMs to generate validation intention-aware test cases, outperforming existing baselines.

Findings

IntentionTest kills 28.1% to 37.6% more common mutants.

It shares 16.9% to 23.9% more common coverage.

It generates 23.7% to 49.0% more successful passing tests.

Abstract

Test cases are valuable assets for maintaining software quality. State-of-the-art automated test generation techniques typically focus on maximizing program branch coverage or translating focal methods into test code. However, in contrast to branch coverage or code-to-test translation, practical tests are written out of the need to validate whether a requirement has been fulfilled. Specifically, each test usually reflects a developer's validation intention for a program function, regarding (1) what is the test scenario of a program function? and (2) what is expected behavior under such a scenario? Without taking such intention into account, generated tests are less likely to be adopted in practice. In this work, we propose IntentionTest, which generates project-specific tests given the description of validation intention. IntentionTest adopts a retrieval-and-edit manner. First, given a focal code and a description of validation intention consisting of a test objective with test precondition and expected results, IntentionTest retrieves a reusable test in the project as the test reference. Then, IntentionTest edits the test reference with an LLM regarding the validation intention toward the target test. We extensively evaluate IntentionTest against four baselines on 3,680 test cases. Compared to state-of-the-art baselines, IntentionTest can (1) generate tests far more semantically relevant to ground-truth tests by (i) killing 28.1% to 37.6% more common mutants and (ii) sharing 16.9% to 23.9% more common coverage; and (2) generate 23.7% to 49.0% more successful passing tests.