Training Data Efficiency in Multimodal Process Reward Models

TL;DR

This paper investigates the data efficiency of training Multimodal Process Reward Models, revealing redundancy in data and proposing a method that achieves full performance with only 10% of the data by prioritizing informative samples.

Contribution

It introduces the Balanced-Information Score (BIS), a novel data selection criterion that improves training efficiency without additional costs, based on theoretical insights into gradient informativeness.

Findings

BIS-selected subsets match or surpass full-data performance.

Using only 10% of data, BIS achieves full performance, outperforming random sampling.

Training saturates quickly under random subsampling, indicating redundancy.

Abstract

Multimodal Process Reward Models (MPRMs) are central to step-level supervision for visual reasoning in MLLMs. Training MPRMs typically requires large-scale Monte Carlo (MC)-annotated corpora, incurring substantial training cost. This paper studies the data efficiency for MPRM training. Our preliminary experiments reveal that MPRM training quickly saturates under random subsampling of the training data, indicating substantial redundancy within existing MC-annotated corpora. To explain this, we formalize a theoretical framework and reveal that informative gradient updates depend on two factors: label mixtures of positive/negative steps and label reliability (average MC scores of positive steps). Guided by these insights, we propose the Balanced-Information Score (BIS), which prioritizes both mixture and reliability based on existing MC signals at the rollout level, without incurring any additional cost. Across two backbones (InternVL2.5-8B and Qwen2.5-VL-7B) on VisualProcessBench, BIS-selected subsets consistently match and even surpass the full-data performance at small fractions. Notably, the BIS subset reaches full-data performance using only 10% of the training data, improving over random subsampling by a relative 4.1%.