PAGENT: Learning to Patch Software Engineering Agents

TL;DR

This paper empirically analyzes the causes of failed patches generated by LLM-based code agents, introduces PAGENT to address type-related errors using static analysis and LLM inference, and demonstrates its effectiveness in fixing such patches.

Contribution

It provides a detailed taxonomy of failure reasons in LLM-generated patches and proposes PAGENT, a novel hybrid approach combining static analysis and LLM inference to improve patch correctness.

Findings

PAGENT fixed 29 out of 127 type-related failed patches.

Seven top LLM code agents produced 769 failed patches across 114 issues.

Failure reasons include incorrect variable type inference and other categories.

Abstract

LLM Agents produce patches automatically to resolve an issue. However, they can generate inaccurate patches. Little is known about the root causes behind those failed patches or how those could be fixed. This paper reports an empirical study of the failed patches generated by seven top LLM code agents. We collected 114 issues from the SWE-bench Lite dataset that remained unresolved across the agents. The seven agents produced a total of 769 failed patches for those issues, which we checked with a combination of GPT-4o and manual analysis. We present a taxonomy of the failure reasons across the patches. The taxonomy contains six categories, with several sub-categories under each category. For example, a frequently observed category is the inability of an LLM to correctly infer/produce the appropriate variable type in the produced patch. As a first step towards addressing such type-related errors, we designed PAGENT (Patch Agent). PAGENT utilizes program analysis techniques like CFG creation and exploration to infer the type of information of a patch. PAGENT does this by applying repository-level static code analysis techniques. Then, PAGENT refines the inferred type by further utilizing an LLM-based inference technique. We tested PAGENT on all 127 type-related failed patches from the top three agents in our study. PAGENT could fix 29 of the 127 failed patches.