ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding

TL;DR

ReST-RL introduces a novel reinforcement learning framework with optimized self-training and decoding techniques to significantly enhance the code reasoning accuracy of large language models, outperforming existing methods on multiple coding benchmarks.

Contribution

The paper presents ReST-RL, a unified RL paradigm combining an improved GRPO algorithm with a value model-assisted decoding method, advancing LLM code reasoning capabilities.

Findings

ReST-RL outperforms baseline methods on coding benchmarks.

Optimized data filtering improves training efficiency.

Decoding with VM-MCTS enhances reasoning accuracy.

Abstract

With respect to improving the reasoning accuracy of LLMs, the representative reinforcement learning (RL) method GRPO faces failure due to insignificant reward variance, while verification methods based on process reward models (PRMs) suffer from difficulties with training data acquisition and verification effectiveness. To tackle these problems, this paper introduces ReST-RL, a unified LLM RL paradigm that significantly improves LLM's code reasoning ability by combining an improved GRPO algorithm with a meticulously designed test time decoding method assisted by a value model (VM). As the first stage of policy reinforcement, ReST-GRPO adopts an optimized ReST algorithm to filter and assemble high-value training data, increasing the reward variance of GRPO sampling, thus improving the effectiveness and efficiency of training. After the basic reasoning ability of LLM policy has been improved, we further propose a test time decoding optimization method called VM-MCTS. Through Monte-Carlo Tree Search (MCTS), we collect accurate value targets with no annotation required, on which VM training is based. When decoding, the VM is deployed by an adapted MCTS algorithm to provide precise process signals as well as verification scores, assisting the LLM policy to achieve high reasoning accuracy. We conduct extensive experiments on coding problems to verify the validity of the proposed RL paradigm. Upon comparison, our approach significantly outperforms other reinforcement training baselines (e.g., naive GRPO and ReST-DPO), as well as decoding and verification baselines (e.g., PRM-BoN and ORM-MCTS) on well-known coding benchmarks of various levels (e.g., APPS, BigCodeBench, and HumanEval), indicating its power to strengthen the reasoning ability of LLM policies. Codes for our project can be found at https://github.com/THUDM/ReST-RL.