StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

TL;DR

StepCodeReasoner introduces explicit intermediate execution supervision and a reinforcement learning algorithm to improve code reasoning accuracy by modeling stepwise execution traces.

Contribution

It presents a novel framework that incorporates intermediate execution states and a bi-level RL method for structured credit assignment, achieving state-of-the-art results.

Findings

Achieved 91.1% on CRUXEval with a 7B model.

Outperformed baselines on LiveCodeBench and REval benchmarks.

Improved code generation performance through explicit execution modeling.

Abstract

Existing code reasoning methods primarily supervise final code outputs, ignoring intermediate states, often leading to reward hacking where correct answers are obtained through inconsistent reasoning. We propose StepCodeReasoner, a framework that introduces explicit intermediate execution-state supervision. By automatically inserting structured print-based execution-trace anchors into code, the model is trained to predict runtime states at each step, transforming code reasoning into a verifiable, stepwise execution modeling problem. Building on this execution-aware method, we introduce Bi-Level GRPO, a reinforcement learning algorithm for structured credit assignment at two levels: inter-trajectory, comparing alternative execution paths, and intra-trajectory, rewarding intermediate accuracy based on its impact on downstream correctness. Extensive experiments demonstrate that StepCodeReasoner achieves SOTA performance in code reasoning. In particular, our 7B model achieves 91.1\% on CRUXEval and 86.5\% on LiveCodeBench, outperforming the CodeReasoner-7B baseline (86.0\% and 77.7\%) and GPT-4o (85.6\% and 75.1\%). Furthermore, on the execution-trace benchmark REval, our model scores 82.9\%, outperforming baseline CodeReasoner-7B (72.3\%), its 14B counterpart (81.1\%), and GPT-4o (77.3\%). Additionally, our approach also improves code generation performance, demonstrating that explicit execution modeling enhances both code reasoning and code generation.