Latent-Space Contrastive Reinforcement Learning for Stable and Efficient LLM Reasoning

TL;DR

This paper introduces DeepLatent Reasoning, a novel latent-space contrastive reinforcement learning framework that improves the stability, efficiency, and scalability of reasoning in large language models by shifting training to a continuous latent space.

Contribution

The paper proposes a latent-space RL approach with a lightweight assistant and contrastive learning to enhance reasoning in LLMs, addressing sample inefficiency and catastrophic forgetting.

Findings

DLR achieves more stable training convergence.

Supports longer-horizon reasoning chains.

Facilitates sustainable accumulation of reasoning capabilities.

Abstract

While Large Language Models (LLMs) demonstrate exceptional performance in surface-level text generation, their nature in handling complex multi-step reasoning tasks often remains one of ``statistical fitting'' rather than systematic logical deduction. Traditional Reinforcement Learning (RL) attempts to mitigate this by introducing a ``think-before-speak'' paradigm. However, applying RL directly in high-dimensional, discrete token spaces faces three inherent challenges: sample-inefficient rollouts, high gradient estimation variance, and the risk of catastrophic forgetting. To fundamentally address these structural bottlenecks, we propose \textbf{DeepLatent Reasoning (DLR)}, a latent-space bidirectional contrastive reinforcement learning framework. This framework shifts the trial-and-error cost from expensive token-level full sequence generation to the continuous latent manifold. Specifically, we introduce a lightweight assistant model to efficiently sample $K$ reasoning chain encodings within the latent space. These encodings are filtered via a dual reward mechanism based on correctness and formatting; only high-value latent trajectories are fed into a \textbf{frozen main model} for single-pass decoding. To maximize reasoning diversity while maintaining coherence, we design a contrastive learning objective to enable directed exploration within the latent space. Since the main model parameters remain frozen during optimization, this method mathematically eliminates catastrophic forgetting. Experiments demonstrate that under comparable GPU computational budgets, DLR achieves more stable training convergence, supports longer-horizon reasoning chains, and facilitates the sustainable accumulation of reasoning capabilities, providing a viable path toward reliable and scalable reinforcement learning for LLMs.