PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification

TL;DR

PatchPilot is a novel rule-based software patching agent that balances efficacy, stability, and cost-efficiency, outperforming existing methods on SWE-bench with low cost and high stability.

Contribution

We introduce PatchPilot, a rule-based patching workflow with novel components and designs that improve patching performance while maintaining low cost and high stability.

Findings

Outperforms existing open-source patching methods on SWE-bench

Maintains low cost under $1 per instance

Ensures higher stability compared to prior approaches

Abstract

Recent research builds various patching agents that combine large language models (LLMs) with non-ML tools and achieve promising results on the state-of-the-art (SOTA) software patching benchmark, SWE-bench. Based on how to determine the patching workflows, existing patching agents can be categorized as agent-based planning methods, which rely on LLMs for planning, and rule-based planning methods, which follow a pre-defined workflow. At a high level, agent-based planning methods achieve high patching performance but with a high cost and limited stability. Rule-based planning methods, on the other hand, are more stable and efficient but have key workflow limitations that compromise their patching performance. In this paper, we propose PatchPilot, an agentic patcher that strikes a balance between patching efficacy, stability, and cost-efficiency. PatchPilot proposes a novel rule-based planning workflow with five components: reproduction, localization, generation, validation, and refinement (where refinement is unique to PatchPilot). We introduce novel and customized designs to each component to optimize their effectiveness and efficiency. Through extensive experiments on the SWE-bench benchmarks, PatchPilot shows a superior performance than existing open-source methods while maintaining low cost (less than 1$ per instance) and ensuring higher stability. We also conduct a detailed ablation study to validate the key designs in each component. Our code is available at https://github.com/ucsb-mlsec/PatchPilot.