PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning

TL;DR

This paper introduces PDCR, a reward framework that decomposes confidence signals into perception and reasoning components to improve vision-language reasoning training.

Contribution

PDCR employs unsupervised skill decomposition and intra-cluster normalization to better align rewards with heterogeneous visual and textual reasoning tasks.

Findings

PDCR outperforms naive global-reward methods on vision-language benchmarks.

Intra-cluster normalization stabilizes confidence signals for perception and reasoning.

Skill decomposition improves the effectiveness of reinforcement learning in V-L reasoning.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal. Recent work shows that providing a fine-grained, model-intrinsic signal (rewarding the confidence growth in the ground-truth answer) effectively improves language reasoning training by providing step-level guidance without costly external models. While effective for unimodal text, we find that naively applying this global reward to vision-language (V-L) reasoning is a suboptimal strategy, as the task is a heterogeneous mix of sparse visual perception and dense textual reasoning. This global normalization creates mixture-induced signal degradation, where the training signal for visual steps is statistically distorted by the predominant textual steps. We propose Perception-Decomposed Confidence Reward (PDCR), a framework that solves this by aligning the reward structure with the task's heterogeneous nature. PDCR first performs an unsupervised skill decomposition, introducing a model-internal Visual Dependence Score to quantify visual reliance and applying a clustering algorithm to separate perception and reasoning steps. Based on this, PDCR computes a decomposed advantage by normalizing confidence gains within each skill cluster. This intra-cluster normalization provides a stable, correctly-scaled signal for both perception and reasoning. We demonstrate that PDCR outperforms the naive, global-reward formulation and sparse-reward baselines on key V-L reasoning benchmarks.