Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair

TL;DR

This paper introduces signal reshaping techniques for GRPO in weak-feedback code repair, significantly improving semantic accuracy and trajectory control in agentic RL settings.

Contribution

It proposes a minimal signal-reshaping construction that enhances GRPO's effectiveness by recovering semantic ranking and localizing credit, leading to substantial performance gains.

Findings

Full signal-reshaped GRPO improves accuracy from 0.385 to 0.535.

Layered rewards and process-score weighting further enhance accuracy and efficiency.

Compared to token-level distillation, the proposed method better captures outcome semantics and control.

Abstract

Code-agent RL often receives weak feedback: rollout-time signals are reliable and executable, but capture only necessary or surface conditions for task success rather than the target semantic predicate. Using agentic compile-fix as the setting, we study signal reshaping for standard GRPO under such feedback. Our central claim is that GRPO's within-group comparison is meaningful only after three kinds of signals are reshaped: outcome rewards recover semantic ranking, process signals localize intra-trajectory credit, and rollouts from the same prompt remain execution-comparable. We operationalize these conditions with a minimal signal-reshaping construction that leaves GRPO's group-normalized advantage construction unchanged: compile-and-semantic layered rewards reshape trajectory ranking, step-level process scores outside group reward normalization reshape within-trajectory update strength, and failure-cause-aware rollout governance reshapes within-group comparability. Experiments show a clear end-to-end gain: full signal-reshaped GRPO improves strict compile-and-semantic accuracy from the base model's zero-shot $0.385$ to $0.535$. Controlled comparisons further explain the source of this gain: binary rewards remove the compile-only middle tier and degrade trajectory control; on top of layered rewards, process-score weighting further improves accuracy from $0.48$ to $0.53$ and reduces average evaluation steps from $23.50$ to $17.02$. As a boundary comparison, privileged-prompt token-level distillation mainly optimizes local distributional alignment; in long tool-use trajectories, this signal is diluted by non-critical tokens and cannot replace outcome semantics, process credit, or within-group comparability.