CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning

TL;DR

CodeReasoner enhances code reasoning in large language models by combining dataset construction, instruction tuning, and reinforcement learning, leading to significant performance improvements over prior methods.

Contribution

It introduces a novel framework integrating dataset creation, instruction tuning, and reinforcement learning to improve code reasoning and generalization in LLMs.

Findings

Improves code reasoning benchmark performance by up to 40.2%.

7B model matches GPT-4o on key tasks.

Scaling to 14B surpasses GPT-4o across benchmarks.

Abstract

Code reasoning is a fundamental capability for large language models (LLMs) in the code domain. It involves understanding and predicting a program's execution behavior, such as determining the output for a given input or whether a specific statement will be executed. This capability is essential for downstream tasks like debugging, code generation, and program repair. Prior approaches mainly rely on supervised fine-tuning to improve performance in code reasoning tasks. However, they often show limited gains and fail to generalize across diverse scenarios. We argue this is due to two core issues: the low quality of training data and the limitations of supervised fine-tuning, which struggles to teach general reasoning skills. To address these challenges, we propose CodeReasoner, a framework that spans both dataset construction and a two-stage training process. First, we introduce a method to construct datasets that focus on the core execution logic of Python programs. Next, we apply instruction tuning to inject execution-specific knowledge distilled from a powerful teacher model. We then enhance reasoning and generalization through GRPO reinforcement learning on top of the fine-tuned model. Experiments on three widely-used code reasoning benchmarks show that CodeReasoner improves performance by 27.1% to 40.2% over prior methods using a 7B model. Notably, the 7B model matches GPT-4o on key tasks like input/output and coverage prediction. When scaled to 14B, CodeReasoner outperforms GPT-4o across all benchmarks. Ablation studies confirm the effectiveness of each training stage and highlight the importance of reasoning chains.