Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learning

TL;DR

This paper introduces Ranking-Aware Calibration (RAC), a training framework for multimodal reinforcement learning that improves model calibration and accuracy by using ranking and corruption-based confidence supervision without external annotations.

Contribution

The paper proposes RAC, a novel training-time calibration method that leverages ranking signals and visual evidence degradation to enhance confidence calibration and reasoning accuracy.

Findings

RAC significantly improves task accuracy across multiple benchmarks.

The pairwise corruption loss reduces calibration error under degraded inputs.

Combining both losses achieves the best calibration and often improves accuracy.

Abstract

Reinforcement learning post-training has substantially improved the reasoning accuracy of vision-language models, yet the resulting policies remain poorly calibrated. Terminal correctness rewards provide no gradient that penalizes confident errors more than uncertain ones and no signal that ties confidence to the quality of visual evidence, a gap that becomes especially severe under corrupted or ambiguous inputs where models continue to report high confidence on incorrect answers. We introduce Ranking-Aware Calibration (RAC), a training-time framework that supervises confidence using two comparison signals that group-based RL already produces at no additional labeling cost. The ranking-aware group loss enforces that a better rollout receives higher confidence than a worse one within the same prompt. The clean--corrupted pairwise loss enforces that confidence attenuates as visual evidence degrades. Because the ranking signal forces the policy to distinguish between correct and incorrect reasoning paths, it also reinforces task accuracy beyond what correctness rewards alone produce. Both losses require no external confidence annotations and integrate naturally with group-based RL post-training. We instantiate RAC on Qwen2.5-VL and InternVL-3.5 backbones and evaluate on six multimodal reasoning benchmarks under clean and corrupted inputs. Empirical results show that the ranking-aware loss substantially improves task accuracy by teaching the policy to discriminate between better and worse reasoning, while the pairwise corruption loss reduces calibration error under degraded inputs. Their combination achieves the best calibration across all tested backbones while improving accuracy in the majority of settings.