SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?

TL;DR

This paper introduces SWE-Mutation, a benchmark for evaluating the quality of LLM-generated test suites in software engineering, revealing current models' limitations in producing reliable and discriminative tests.

Contribution

It presents a novel benchmark and an agentic mutation framework to systematically assess and improve the realism and effectiveness of LLM-generated test suites.

Findings

Current LLMs achieve low verification and detection rates.

Agentic mutation reduces detection rates, increasing test suite realism.

The benchmark includes 2,636 mutants across nine programming languages.

Abstract

Evaluating software engineering capabilities has become a core component of modern large language models (LLMs); however, the key bottleneck hindering further scaling lies not in the scarcity of high-quality solutions, but in the lack of high-quality test suites. Test suites are indispensable both for synthesizing program repair trajectories and for providing precise feedback signals in reinforcement learning. Unfortunately, due to the high cost and difficulty of annotation, high-quality test suites have long been hard to obtain, while those automatically generated by LLMs tend to be superficial and lack sufficient discriminative power. As a first step toward constructing high-quality test suites, we introduce SWE-Mutation, a benchmark for evaluating LLM-generated test suites. The benchmark characterizes test suites by introducing systematically mutated solutions that attempt to ``fool'' the test suites and pass validation. We further propose an agentic, language-agnostic framework for automatically generating complex mutants. Our benchmark consists of 2,636 mutated variants derived from 800 original instances and includes a multilingual subset spanning nine programming languages. Experiments on seven LLMs reveal that even DeepSeek-V3.1 achieves only 10.20% verification and 36.15% detection rates, highlighting the inadequacy of current LLMs. Additionally, our agentic mutation strategy enhances realism, reducing average detection rates from 71.04% to 39.81% compared to conventional methods. These findings expose persistent deficiencies in the ability of current LLMs to generate reliable and discriminative test suites.