Summary-Mediated Repair: Can LLMs use code summarisation as a tool for program repair?

TL;DR

This paper introduces summary-mediated repair, a novel approach using code summarisation as an intermediate step to improve program repair in large language models, showing modest but consistent gains across multiple models and benchmarks.

Contribution

The paper proposes a prompt-only pipeline that leverages natural-language code summarisation for program repair, extending previous work on summarisation as an intermediary for downstream tasks.

Findings

Error-aware summaries repair up to 65% of unseen errors.

Summaries improve repair performance by an average of 5% over baseline.

The approach is effective across eight production-grade LLMs and two benchmarks.

Abstract

Large Language Models (LLMs) often produce code with subtle implementation-level bugs despite strong benchmark performance. These errors are hard for LLMs to spot and can have large behavioural effects; yet when asked to summarise code, LLMs can frequently surface high-level intent and sometimes overlook this low-level noise. Motivated by this, we propose summary-mediated repair, a prompt-only pipeline for program repair that leverages natural-language code summarisation as an explicit intermediate step, extending previous work that has already shown code summarisation to be a useful intermediary for downstream tasks. We evaluate our method across eight production-grade LLMs on two function level benchmarks (HumanEvalPack and MBPP), comparing several summary styles against a direct repair baseline. Error-aware diagnostic summaries consistently yield the largest gains - repairing up to 65% of unseen errors, on average of 5% more than the baseline - though overall improvements are modest and LLM-dependent. Our results position summaries as a cheap, human-interpretable diagnostic artefact that can be integrated into program-repair pipelines rather than a stand-alone fix-all.