Beyond Accuracy: Behavioral Dynamics of Agentic Multi-Hunk Repair

TL;DR

This study systematically evaluates large language model-driven coding agents on multi-hunk bug repair, revealing their strengths, limitations, and the impact of context-aware tools like Maple on repair accuracy.

Contribution

First comprehensive analysis of LLM-driven agents on multi-hunk bug repair, introducing fine-grained metrics and the Maple tool for improved localization and repair performance.

Findings

Repair accuracy varies significantly among agents.

Higher bug dispersion reduces repair success.

Maple improves Gemini-cli's accuracy by 30%.

Abstract

Automated program repair has traditionally focused on single-hunk defects, overlooking multi-hunk bugs that are prevalent in real-world systems. Repairing these bugs requires coordinated edits across multiple, disjoint code regions, posing substantially greater challenges. We present the first systematic study of LLM-driven coding agents (Claude Code, Codex, Gemini-cli, and Qwen Code) on this task. We evaluate these agents on 372 multi-hunk bugs from the Hunk4J dataset, analyzing 1,488 repair trajectories using fine-grained metrics that capture localization, repair accuracy, regression behavior, and operational dynamics. Results reveal substantial variation: repair accuracy ranges from 25.8% (Qwen Code) to 93.3% (Claude Code) and consistently declines with increasing bug dispersion and complexity. High-performing agents demonstrate superior semantic consistency, achieving positive regression reduction, whereas lower-performing agents often introduce new test failures. Notably, agents do not fail fast; failed repairs consume substantially more resources (39%-343% more tokens) and require longer execution time (43%-427%). Additionally, we developed Maple to provide agents with repository-level context. Empirical results show that Maple improves the repair accuracy of Gemini-cli by 30% through enhanced localization. By analyzing fine-grained metrics and trajectory-level analysis, this study moves beyond accuracy to explain how coding agents localize, reason, and act during multi-hunk repair.