VeRPO: Verifiable Dense Reward Policy Optimization for Code Generation

TL;DR

VeRPO introduces a verifiable, dense reward framework for code generation in reinforcement learning, improving performance by dynamically weighting partial success signals from unit tests, leading to more efficient and reliable training.

Contribution

The paper presents VeRPO, a novel RL method that constructs dense, verifiable rewards from execution feedback, addressing reward sparsity and misalignment issues in code generation.

Findings

Outperforms baselines with up to +8.83% pass@1 improvement.

Achieves this with negligible time and memory overhead.

Demonstrates robustness across diverse benchmarks.

Abstract

Effective reward design is a central challenge in Reinforcement Learning (RL) for code generation. Mainstream pass/fail outcome rewards enforce functional correctness via executing unit tests, but the resulting sparsity limits potential performance gains. While recent work has explored external Reward Models (RM) to generate richer, continuous rewards, the learned RMs suffer from reward misalignment and prohibitive computational cost. In this paper, we introduce \textbf{VeRPO} (\textbf{V}erifiable D\textbf{e}nse \textbf{R}eward \textbf{P}olicy \textbf{O}ptimization), a novel RL framework for code generation that synthesizes \textit{robust and dense rewards fully grounded in verifiable execution feedback}. The core idea of VeRPO is constructing dense rewards from weighted partial success: by dynamically estimating the difficulty weight of each unit test based on the execution statistics during training, a dense reward is derived from the sum of weights of the passed unit tests. To solidify the consistency between partial success and end-to-end functional correctness, VeRPO further integrates the dense signal with global execution outcomes, establishing a robust and dense reward paradigm relying solely on verifiable execution feedback. Extensive experiments across diverse benchmarks and settings demonstrate that VeRPO consistently outperforms outcome-driven and RM-based baselines, achieving up to +8.83\% gain in pass@1 with negligible time cost (< 0.02\%) and zero GPU memory overhead.