Coupled Variational Reinforcement Learning for Language Model General Reasoning

TL;DR

This paper introduces CoVRL, a novel reinforcement learning framework that couples variational inference with RL to improve reasoning in language models, achieving significant performance gains on reasoning benchmarks.

Contribution

It proposes a coupled variational RL method that enhances exploration and coherence in reasoning traces, advancing verifier-free RL for language models.

Findings

Improves reasoning performance by 12.4% over base models

Achieves 2.3% higher accuracy than state-of-the-art verifier-free RL methods

Provides a new principled framework for reasoning in language models

Abstract

While reinforcement learning has achieved impressive progress in language model reasoning, it is constrained by the requirement for verifiable rewards. Recent verifier-free RL methods address this limitation by utilizing the probabilities that LLMs generate reference answers as reward signals. However, these approaches typically sample reasoning traces conditioned only on the question. This design decouples reasoning-trace sampling from answer information, leading to inefficient exploration and incoherence between traces and final answers. In this paper, we propose \textit{\b{Co}upled \b{V}ariational \b{R}einforcement \b{L}earning} (CoVRL), which bridges variational inference and reinforcement learning by coupling prior and posterior distributions through a hybrid sampling strategy. By constructing and optimizing a composite distribution that integrates these two distributions, CoVRL enables efficient exploration while preserving strong thought-answer coherence. Extensive experiments on mathematical and general reasoning benchmarks show that CoVRL improves performance by 12.4\% over the base model and achieves an additional 2.3\% improvement over state-of-the-art verifier-free RL baselines, providing a principled framework for enhancing the general reasoning capabilities of language models.