Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models

TL;DR

This paper introduces Proxy-GRM, a reinforcement learning approach that trains lightweight proxy agents to generate high-quality, transferable rubrics for vision-language models, significantly improving reward evaluation accuracy and transferability.

Contribution

It proposes Proxy-GRM, a novel method that explicitly optimizes rubrics via proxy-guided verification, enhancing their quality and transferability in VLM reward models.

Findings

Proxy-GRM achieves state-of-the-art results on multiple reward benchmarks.

Proxy rubrics transfer effectively to unseen evaluators, improving test-time reward accuracy.

Proxy-SFT outperforms Proxy-RL as a verifier in rubric quality.

Abstract

Generative reward models (GRMs) for vision-language models (VLMs) often evaluate outputs via a three-stage pipeline: rubric generation, criterion-based scoring, and a final verdict. However, the intermediate rubric is rarely optimized directly. Prior work typically either treats rubrics as incidental or relies on expensive LLM-as-judge checks that provide no differentiable signal and limited training-time guidance. We propose Proxy-GRM, which introduces proxy-guided rubric verification into Reinforcement Learning (RL) to explicitly enhance rubric quality. Concretely, we train lightweight proxy agents (Proxy-SFT and Proxy-RL) that take a candidate rubric together with the original query and preference pair, and then predict the preference ordering using only the rubric as evidence. The proxy's prediction accuracy serves as a rubric-quality reward, incentivizing the model to produce rubrics that are internally consistent and transferable. With ~50k data samples, Proxy-GRM reaches state-of-the-art results on the VL-Reward Bench, Multimodal Reward Bench, and MM-RLHF-Reward Bench, outperforming the methods trained on four times the data. Ablations show Proxy-SFT is a stronger verifier than Proxy-RL, and implicit reward aggregation performs best. Crucially, the learned rubrics transfer to unseen evaluators, improving reward accuracy at test time without additional training. Our code is available at https://github.com/Qwen-Applications/Proxy-GRM.