Stabilizing Long-term Multi-turn Reinforcement Learning with Gated Rewards

TL;DR

This paper introduces a new reward shaping method called Gated Reward Accumulation (G-RA) within a unified RL framework to improve long-term multi-turn reinforcement learning in software engineering tasks, addressing reward sparsity and misalignment issues.

Contribution

The paper proposes G-RA, a novel reward accumulation technique that enhances stability and performance in long-horizon RL, along with a comprehensive SWE-oriented RL framework supporting multi-turn interactions.

Findings

G-RA significantly increases task completion rates.

G-RA improves modification rates without policy degradation.

The framework effectively supports multi-turn reasoning in SWE tasks.

Abstract

Reward sparsity in long-horizon reinforcement learning (RL) tasks remains a significant challenge, while existing outcome-based reward shaping struggles to define meaningful immediate rewards without introducing bias or requiring explicit task decomposition. Alternatively, verification-based reward shaping uses stepwise critics, but misalignment between immediate rewards and long-term objectives can lead to reward hacking and suboptimal policies. In this work, we address this problem in the context of software engineering (SWE) tasks, where multi-turn reasoning and rule-based verification are critical. We introduce the SWE-oriented RL Framework, a unified system supporting multi-turn interaction, docker-based execution, and customizable reward functions. Additionally, we propose Gated Reward Accumulation (G-RA), a novel method that accumulates immediate rewards only when high-level (long-term) rewards meet a predefined threshold, ensuring stable RL optimization. Experiments on SWE-bench Verified and kBench demonstrate that G-RA leads to an increase in completion rates (47.6\% \rightarrow 93.8\% and 22.0\% \rightarrow 86.0\%) and modification rates (19.6\% \rightarrow 23.8\% and 12.0\% \rightarrow 42.0\%), while avoiding policy degradation caused by reward misalignment. Our findings highlight the importance of balanced reward accumulation in long-horizon RL and provide a practical solution.