MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning

TL;DR

This paper introduces MSRL, a multi-stage reinforcement learning framework that enables scalable training of generative multimodal reward models using limited multimodal data, significantly improving performance without extra annotations.

Contribution

MSRL proposes a novel multi-stage reinforcement learning approach that transfers textual reward reasoning to multimodal tasks, reducing reliance on costly preference data.

Findings

Improved performance on VL-RewardBench from 66.6% to 75.9%.

Enhanced results on GenAI-Bench from 70.2% to 75.7%.

Effective scaling of multimodal reward modeling without additional annotations.

Abstract

Recent advances in multimodal reward modeling have been largely driven by a paradigm shift from discriminative to generative approaches. Building on this progress, recent studies have further employed reinforcement learning from verifiable rewards (RLVR) to enhance multimodal reward models (MRMs). Despite their success, RLVR-based training typically relies on labeled multimodal preference data, which are costly and labor-intensive to obtain, making it difficult to scale MRM training. To overcome this limitation, we propose a Multi-Stage Reinforcement Learning (MSRL) approach, which can achieve scalable RL for MRMs with limited multimodal data. MSRL replaces the conventional RLVR-based training paradigm by first learning a generalizable reward reasoning capability from large-scale textual preference data, and then progressively transferring this capability to multimodal tasks through caption-based and fully multimodal reinforcement-learning stages. Furthermore, we introduce a cross-modal knowledge distillation approach to improve preference generalization within MSRL. Extensive experiments demonstrate that MSRL effectively scales the RLVR-based training of generative MRMs and substantially improves their performance across both visual understanding and visual generation tasks (e.g., from 66.6% to 75.9% on VL-RewardBench and from 70.2% to 75.7% on GenAI-Bench), without requiring additional multimodal preference annotations. Our code is available at: https://github.com/wangclnlp/MSRL.