Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback

TL;DR

This paper introduces Property-Generated Solver (PGS), a feedback mechanism for LLM code refinement that emphasizes high-quality, property-oriented, and minimal feedback to improve correctness and fix rates.

Contribution

The paper proposes PGS, a novel feedback paradigm that enhances LLM code refinement by focusing on semantic properties and minimal counterexamples, outperforming existing methods.

Findings

PGS improves pass@1 by up to 13.4% over other TDD methods.

Achieves over 64% fix rate on initially failed problems.

Demonstrates a bug fix rate 1.4x-1.6x higher than previous approaches.

Abstract

LLMs excel at code generation, yet ensuring the functional correctness of their outputs remains a persistent challenge. While recent studies have applied Test-Driven Development (TDD) to refine code, these methods are often undermined by poor feedback quality, stemming from the scarcity of high-quality test cases and noisy signals from auto-generated ones. In this work, we shift the focus from test quantity to feedback quality. We introduce the Property-Generated Solver (PGS), a novel paradigm designed to generate highly effective feedback via two principles: it must be property-oriented, to provide semantic guidance beyond simple I/O mismatches, and structurally minimal, to reduce cognitive load and isolate root causes. PGS operates by checking high-level program properties (e.g., a sorting function must produce a non-decreasing sequence) then providing the simplest failing counterexample to the LLM. By adhering to these principles, this targeted feedback mechanism leads to significant performance gains. Specifically, PGS achieves an improvement of up to 13.4% in pass@1 against other TDD-based methods and an over 64% fix rate on problems where the model initially failed. This property-driven, minimal feedback steers LLMs toward correct and generalizable solutions. Across diverse benchmarks, PGS demonstrates superior performance, achieving a bug fix rate 1.4x-1.6x higher than the strongest debugging-based approaches and establishing a new state-of-the-art in automated code refinement.