Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning

TL;DR

Skywork-VL Reward is a new multimodal reward model that enhances understanding and reasoning in vision-language tasks by leveraging a large preference dataset and a specialized architecture, achieving state-of-the-art results.

Contribution

The paper introduces Skywork-VL Reward, a novel multimodal reward model with a large-scale preference dataset and a multi-stage fine-tuning approach, advancing multimodal alignment capabilities.

Findings

Achieves state-of-the-art results on VL-RewardBench.

Demonstrates competitive performance on text-only RewardBench.

Preference data improves multimodal reasoning through MPO.

Abstract

We propose Skywork-VL Reward, a multimodal reward model that provides reward signals for both multimodal understanding and reasoning tasks. Our technical approach comprises two key components: First, we construct a large-scale multimodal preference dataset that covers a wide range of tasks and scenarios, with responses collected from both standard vision-language models (VLMs) and advanced VLM reasoners. Second, we design a reward model architecture based on Qwen2.5-VL-7B-Instruct, integrating a reward head and applying multi-stage fine-tuning using pairwise ranking loss on pairwise preference data. Experimental evaluations show that Skywork-VL Reward achieves state-of-the-art results on multimodal VL-RewardBench and exhibits competitive performance on the text-only RewardBench benchmark. Furthermore, preference data constructed based on our Skywork-VL Reward proves highly effective for training Mixed Preference Optimization (MPO), leading to significant improvements in multimodal reasoning capabilities. Our results underscore Skywork-VL Reward as a significant advancement toward general-purpose, reliable reward models for multimodal alignment. Our model has been publicly released to promote transparency and reproducibility.