Consistency Meets Verification: Enhancing Test Generation Quality in Large Language Models Without Ground-Truth Solutions

TL;DR

ConVerTest is a new two-stage pipeline that improves the quality and reliability of test generation in large language models without needing ground-truth solutions, by combining self-consistency, iterative verification, and cross-validation strategies.

Contribution

It introduces ConVerTest, a novel approach that enhances test validity and coverage in LLMs without relying on prior code, addressing hallucinations and verification challenges.

Findings

Up to 39% improvement in test validity.

Up to 28% increase in line coverage.

Up to 18% higher mutation scores.

Abstract

Large Language Models (LLMs) have significantly advanced automated test generation, yet existing methods often rely on ground-truth code for verification, risking bug propagation and limiting applicability in test-driven development. We present ConVerTest, a novel two-stage pipeline for synthesizing reliable tests without requiring prior code implementations. ConVerTest integrates three core strategies: (i) Self-Consistency(SC) to generate convergent test cases via majority voting; (ii) Chain-of-Verification (CoVe) for iterative, reasoning-guided code refinement; and (iii) a Dual Execution Agreement to crossvalidate code and tests through consensus. Experiments on BIGCODEBENCH and LESS BASIC PYTHON PROBLEMS (LBPP) benchmarks demonstrate that ConVerTest improves test validity, line coverage, and mutation scores by up to 39%, 28%, and 18% respectively over baselines. Our findings highlight ConVerTest as a robust solution for mitigating hallucinations and enhancing the reliability of autonomous software testing agents.