Multi-Path Collaborative Reasoning via Reinforcement Learning

TL;DR

This paper introduces M3PO, a reinforcement learning framework that enhances reasoning diversity and reliability in large language models by enabling multi-path collaborative reasoning, leading to state-of-the-art results.

Contribution

M3PO is a novel RL-based approach that promotes diverse reasoning paths and peer feedback integration, improving reasoning robustness and performance.

Findings

Achieves state-of-the-art on reasoning benchmarks

Maintains interpretability and inference efficiency

Enhances multi-step reasoning reliability

Abstract

Chain-of-Thought (CoT) reasoning has significantly advanced the problem-solving capabilities of Large Language Models (LLMs), yet conventional CoT often exhibits internal determinism during decoding, limiting exploration of plausible alternatives. Recent methods attempt to address this by generating soft abstract tokens to enable reasoning in a continuous semantic space. However, we find that such approaches remain constrained by the greedy nature of autoregressive decoding, which fundamentally isolates the model from alternative reasoning possibilities. In this work, we propose Multi-Path Perception Policy Optimization (M3PO), a novel reinforcement learning framework that explicitly injects collective insights into the reasoning process. M3PO leverages parallel policy rollouts as naturally diverse reasoning sources and integrates cross-path interactions into policy updates through a lightweight collaborative mechanism. This design allows each trajectory to refine its reasoning with peer feedback, thereby cultivating more reliable multi-step reasoning patterns. Empirical results show that M3PO achieves state-of-the-art performance on both knowledge- and reasoning-intensive benchmarks. Models trained with M3PO maintain interpretability and inference efficiency, underscoring the promise of multi-path collaborative learning for robust reasoning.