ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning

TL;DR

ExecVerify introduces a reinforcement learning approach with verifiable stepwise rewards for code execution reasoning, significantly improving smaller models' performance on reasoning and code generation benchmarks.

Contribution

The paper presents a novel white-box reward framework and a two-stage training pipeline that enhances code execution reasoning in smaller language models.

Findings

7B model with ExecVerify matches 32B model performance on reasoning benchmarks.

Up to 5.9% improvement in pass@1 on code generation tasks.

Effective reinforcement learning with verifiable rewards enhances reasoning accuracy.

Abstract

Code LLMs still struggle with code execution reasoning, especially in smaller models. Existing methods rely on supervised fine-tuning (SFT) with teacher-generated explanations, primarily in two forms: (1) input-output (I/O) prediction chains and (2) natural-language descriptions of execution traces. However, intermediate execution steps cannot be explicitly verified during SFT, so the training objective can reduce to merely matching teacher explanations. Moreover, training data is typically collected without explicit control over task difficulty. We introduce ExecVerify, which goes beyond text imitation by incorporating verifiable white-box rewards derived from execution traces, including next-statement prediction and variable value/type prediction. Our work first builds a dataset with multiple difficulty levels via constraint-based program synthesis. Then, we apply reinforcement learning (RL) to reward correct answers about both intermediate execution steps and final outputs, aligning the training objective with semantic correctness at each execution step. Finally, we adopt a two-stage training pipeline that first enhances execution reasoning and then transfers to code generation. Experiments demonstrate that a 7B model trained with ExecVerify achieves performance comparable to 32B models on code reasoning benchmarks and improves pass@1 by up to 5.9\% on code generation tasks over strong post-training baselines.