GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning

TL;DR

GM-PRM is a novel generative model that actively collaborates with multimodal language models to improve complex mathematical reasoning by providing detailed analysis and corrections, leading to state-of-the-art results.

Contribution

Introduces GM-PRM, a generative, interpretable process reward model that actively corrects reasoning errors and enhances multimodal mathematical reasoning performance.

Findings

Achieves state-of-the-art results on multimodal math benchmarks.

Requires only 20K training samples for effective performance.

Provides detailed, step-level interpretability and correction capabilities.

Abstract

Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities but often struggle with complex, multi-step mathematical reasoning, where minor errors in visual perception or logical deduction can lead to complete failure. While Process Reward Models (PRMs) offer step-by-step supervision, existing multimodal PRMs are limited to being binary verifiers that can identify but not correct errors, offering little explanatory power. To address these deficiencies, we introduce the Generative Multimodal Process Reward Model (GM-PRM), a novel paradigm that transforms the PRM from a passive judge into an active reasoning collaborator. Instead of a simple scalar score, GM-PRM provides a fine-grained, interpretable analysis of each reasoning step, evaluating its step intent, visual alignment, and logical soundness. More critically, GM-PRM is trained to generate a corrected version of the first erroneous step it identifies. This unique corrective capability enables our new test-time inference strategy, Refined Best-of-N (Refined-BoN). This framework actively enhances solution quality by using the PRM's generated correction to guide the policy model toward a more promising reasoning trajectory, thereby improving the diversity and correctness of the solution pool. We demonstrate that GM-PRM achieves state-of-the-art results on multiple multimodal math benchmarks, significantly boosting policy model performance with remarkable data efficiency, requiring only a 20K-sample training dataset. Our code will be released upon acceptance.