Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models

TL;DR

This paper introduces Joint Reward Modeling (JRM), a method that combines preference learning and language modeling to create efficient, semantically rich reward models for complex visual tasks, outperforming existing approaches.

Contribution

The paper proposes a novel joint training framework that internalizes generative reasoning into discriminative reward models, enhancing both efficiency and semantic understanding.

Findings

Achieves state-of-the-art results on MMRB2 and EditReward-Bench.

Improves stability and performance in downstream reinforcement learning.

Effectively bridges efficiency and semantic understanding in reward modeling.

Abstract

Reward models are critical for reinforcement learning from human feedback, as they determine the alignment quality and reliability of generative models. For complex tasks such as image editing, reward models are required to capture global semantic consistency and implicit logical constraints beyond local similarity. Existing reward modeling approaches have clear limitations. Discriminative reward models align well with human preferences but struggle with complex semantics due to limited reasoning supervision. Generative reward models offer stronger semantic understanding and reasoning, but they are costly at inference time and difficult to align directly with human preferences. To this end, we propose Joint Reward Modeling (JRM), which jointly optimizes preference learning and language modeling on a shared vision-language backbone. This approach internalizes the semantic and reasoning capabilities of generative models into efficient discriminative representations, enabling fast and accurate evaluation. JRM achieves state-of-the-art results on MMRB2 and EditReward-Bench, and significantly improves stability and performance in downstream online reinforcement learning. These results show that joint training effectively bridges efficiency and semantic understanding in reward modeling.