Echo: Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation

TL;DR

Echo is an innovative agent that enhances bug reproduction test generation by leveraging code graphs, automatic execution, and patch validation, significantly improving success rates over previous methods.

Contribution

It introduces a novel approach combining graph-based context retrieval, automatic test execution, and patch validation for bug reproduction, surpassing prior tools in effectiveness.

Findings

Achieves a 66.28% success rate on SWT-Bench Verified.

Improves context retrieval with a code graph and query refinement.

Automatically executes generated tests and validates patches.

Abstract

Identifying the root cause of a bug remains difficult for many developers because bug reports often lack a bug reproducing test case that reliably triggers the failure. Manually writing such test cases is time-consuming and requires substantial effort to understand the codebase and isolate the failing behavior. To address this challenge, we propose Echo, an agent for generating issue reproducing test cases, which advances previous work in several ways. During generation, Echo strengthens context retrieval by leveraging a code graph and a novel automatic query-refinement strategy. Echo also improves upon previous tools by automatically executing generated test cases, a first-of-its-kind feature that seamlessly integrates into practical development workflows. In addition, Echo generates potential patches and uses the patched version to validate whether a candidate test meets the fail-to-pass criterion and to provide actionable feedback for refinement. Unlike prior bug-reproduction agents that sample and rank multiple candidate tests, Echo generates a single test per issue, offering a better cost-performance trade-off. Experiments on SWT-Bench Verified show that Echo establishes a new state of the art among open-source approaches, achieving a 66.28% success rate.