Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

TL;DR

This paper investigates the role and value of agent-generated tests in LLM-based software engineering, finding that such tests mainly serve observational purposes and do not significantly impact task success.

Contribution

The study provides empirical evidence that current agent-written tests are more about process than improving final outcomes, challenging assumptions about their utility.

Findings

Test writing is common across models but does not differ significantly between resolved and unresolved tasks.

Agent-written tests mainly serve as observational feedback, with print statements being more common than assertions.

Prompt modifications to increase or decrease test writing do not significantly affect final task outcomes.

Abstract

Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. For example, GPT-5.2 writes almost no new tests yet achieves performance comparable to top-ranking agents.This raises a central question: do such tests meaningfully improve issue resolution, or do they mainly mimic a familiar software-development practice while consuming interaction budget? To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.