Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning

TL;DR

This paper introduces CodeThinker, a reinforcement learning framework that enhances large language models' code reasoning by leveraging stepwise consistency, resulting in improved accuracy and robustness across benchmarks and downstream tasks.

Contribution

It proposes a novel consistency-driven reinforcement learning approach with a reasoning-aware training module, dynamic sampling, and a reward mechanism to improve code reasoning in LLMs.

Findings

Achieves state-of-the-art accuracy, outperforming baselines by 4.3% on Qwen2.5-Coder-7B-Instruct.

Gains 5.33% and 3.11% accuracy on mathematical and code reasoning tasks across 17 languages.

Effectively alleviates reward hacking and enhances reasoning capabilities.

Abstract

Code reasoning refers to the task of predicting the output of a program given its source code and specific inputs. It can measure the reasoning capability of large language models (LLMs) and also benefit downstream tasks such as code generation and mathematical reasoning. Existing work has verified the effectiveness of reinforcement learning on the task. However, these methods design rewards solely based on final outputs or coarse-grained signals, and neglect the inherent consistency of the stepwise reasoning process in the task. Therefore, these methods often result in sparse reward or reward hacking, which limits the full play of enhanced learning capabilities. To alleviate these issues, we propose CodeThinker, a consistency-driven reinforcement learning framework for code reasoning. Specifically, CodeThinker has three key components: (1) a stepwise reasoning-aware model training module, which utilizes a consistency tracing paradigm as a template to synthesize training data that captures the stepwise reasoning process; (2) a dynamic beam sampling strategy, which aims to improve the quality of sampled outputs under a fixed sampling budget; and (3) a consistency reward mechanism that can effectively alleviate reward hacking. Experiments on three popular benchmarks show that CodeThinker achieves state-of-the-art performance across multiple LLMs. For instance, it outperforms the strongest baseline by 4.3% in accuracy when deployed on Qwen2.5-Coder-7B-Instruct. We also validate the effectiveness of CodeThinker on downstream tasks. Results show that, without additional training, CodeThinker obtains average accuracy gains of 5.33 and 3.11 percentage points on mathematical reasoning and code reasoning tasks covering 17 programming languages, respectively.