Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

TL;DR

This paper introduces POW3R, a policy-aware rubric reward framework that dynamically adjusts criterion weights during reinforcement learning to improve policy training efficiency and effectiveness.

Contribution

POW3R preserves human rubric importance while adaptively emphasizing criteria that distinguish current policy outputs, enhancing training signals in RL with rubric rewards.

Findings

POW3R outperforms vanilla GRPO in 24 of 30 policy/metric comparisons.

POW3R improves mean rubric reward and strict completion rates.

Training with POW3R reaches performance plateaus 2.5 to 4 times faster.

Abstract

Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins $24$ of $30$ base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in $2.5$--$4\times$ fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.