Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization

TL;DR

This paper introduces Manifold-Reshaping Policy Optimization (MRPO), a geometric approach that expands the reasoning capabilities of large language models by restructuring their inference space, leading to state-of-the-art performance on mathematical tasks.

Contribution

The paper presents MRPO, a novel geometric framework that fundamentally reshapes the inference space of LLMs, enabling enhanced reasoning beyond traditional alignment methods.

Findings

MRPO outperforms larger models on mathematical reasoning tasks.

It significantly expands the reasoning capacity boundary of LLMs.

The approach achieves state-of-the-art results with a 4B-parameter model.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). However, recent studies question whether RL genuinely expands reasoning capacity or merely aligns existing latent capabilities, arguing that exploration remains confined within the pre-trained model's low-rank bias manifold. In this work, we challenge this accessibility boundary hypothesis by demonstrating that the latent reasoning space can be fundamentally expanded through targeted geometric interventions. We propose Manifold-Reshaping Policy Optimization (MRPO), a geometric framework designed to fundamentally restructure the inference space of LLMs. MRPO operates in two stages: first, we employ Spectral Orthogonal Exploration (SOE) to eject the policy initialization into the null space of the bias manifold; second, we integrate an Effective Rank regularization term into the policy optimization objective. This approach incentivizes the discovery and maintenance of high-dimensional reasoning trajectories against the entropy-reducing tendency of standard RL. Empirically, our 4B-parameter method achieves state-of-the-art performance on mathematical tasks, significantly outperforming larger models (e.g., Qwen3-32B) and expanding the capability boundary beyond standard GRPO. Our code is available at https://anonymous.4open.science/r/MRPO-D57B/