PROPA: Toward Process-level Optimization in Visual Reasoning via Reinforcement Learning

TL;DR

PROPA introduces a process-level optimization framework combining Monte Carlo Tree Search and reinforcement learning to enhance multi-step visual reasoning in vision-language models, achieving significant performance improvements.

Contribution

It presents a novel framework that integrates MCTS with GRPO for dense, process-level rewards, enabling stable, step-wise reasoning optimization without human annotations.

Findings

Up to 17.0% gains on in-domain tasks

Up to 21.0% gains on out-of-domain tasks

Consistent outperformance over baselines across benchmarks

Abstract

Despite significant progress, Vision-Language Models (VLMs) still struggle with complex visual reasoning, where multi-step dependencies cause early errors to cascade through the reasoning chain. Existing post-training paradigms are limited: Supervised Fine-Tuning (SFT) relies on costly step-level annotations, while Reinforcement Learning with Verifiable Rewards (RLVR) methods like GRPO provide only sparse, outcome-level feedback, hindering stable optimization. We introduce PROPA (Process-level Reasoning Optimization with interleaved Policy Alignment), a novel framework that integrates Monte Carlo Tree Search (MCTS) with GRPO to generate dense, process-level rewards and optimize reasoning at each intermediate step without human annotations. To overcome the cold-start problem, PROPA interleaves GRPO updates with SFT, enabling the model to learn from both successful and failed reasoning trajectories. A Process Reward Model (PRM) is further trained to guide inference-time search, aligning the test-time search with the training signal. Across seven benchmarks and four VLM backbones, PROPA consistently outperforms both SFT- and RLVR-based baselines. It achieves up to 17.0% gains on in-domain tasks and 21.0% gains on out-of-domain tasks compared to existing state-of-the-art, establishing a strong reasoning and generalization capability for visual reasoning tasks. The code isavailable at: https://github.com/YanbeiJiang/PROPA.