Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning

TL;DR

Latent-GRPO introduces a novel reinforcement learning method for latent reasoning that overcomes stability issues, leading to more efficient and accurate reasoning with shorter chains across various benchmarks.

Contribution

It proposes Latent-GRPO, a new algorithm that addresses key challenges in latent RL reasoning, improving stability and performance over existing methods.

Findings

Latent-GRPO improves Pass@1 by 7.86 points on low-difficulty tasks.

It surpasses explicit GRPO by 4.27 points on high-difficulty tasks.

Achieves stronger pass@$k$ performance with shorter reasoning chains.

Abstract

Latent reasoning offers a more efficient alternative to explicit reasoning by compressing intermediate reasoning into continuous representations and substantially shortening reasoning chains. However, existing latent reasoning methods mainly focus on supervised learning, and reinforcement learning in latent space remains highly unstable. We study this problem through the lens of Group Relative Policy Optimization (GRPO), and show that directly adapting GRPO to latent reasoning is fundamentally non-trivial: latent reasoning changes both the probability density and the sampling mechanism, causing three coupled bottlenecks: absence of intrinsic latent manifolds, where unconstrained exploration pushes rollouts off the valid latent manifold; exploration-optimization misalignment, where trajectory-level rewards can induce incorrect token-level updates; and latent mixture non-closure, where jointly reinforcing multiple correct latent paths can produce an invalid averaged state. To address them, we propose \textbf{Latent-GRPO}, which combines invalid-sample advantage masking, one-sided noise sampling, and optimal correct-path first-token selection. Across four low-difficulty benchmarks (e.g., GSM8K-Aug) and four high-difficulty benchmarks (e.g., AIME), Latent-GRPO improves over its latent initialization by 7.86 Pass@1 points on low-difficulty tasks and surpasses explicit GRPO by 4.27 points on high-difficulty tasks while using 3--4$\times$ shorter reasoning chains. It also achieves stronger pass@$k$ performance under Gumbel sampling. These results establish Latent-GRPO as an effective approach for stable and efficient latent reasoning.