VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation

TL;DR

VisionReward introduces a hierarchical, interpretable framework for learning fine-grained human preferences in image and video generation, significantly improving reward accuracy and alignment with human judgments.

Contribution

It presents a novel multi-dimensional, interpretable reward model for visual generative tasks, addressing limitations of black-box approaches and enhancing preference learning.

Findings

Outperforms existing reward models in preference prediction accuracy by 17.2%.

Achieves a 31.6% higher pairwise win rate in text-to-video generation.

Demonstrates significant improvements in aligning generated content with human preferences.

Abstract

Visual generative models have achieved remarkable progress in synthesizing photorealistic images and videos, yet aligning their outputs with human preferences across critical dimensions remains a persistent challenge. Though reinforcement learning from human feedback offers promise for preference alignment, existing reward models for visual generation face limitations, including black-box scoring without interpretability and potentially resultant unexpected biases. We present VisionReward, a general framework for learning human visual preferences in both image and video generation. Specifically, we employ a hierarchical visual assessment framework to capture fine-grained human preferences, and leverages linear weighting to enable interpretable preference learning. Furthermore, we propose a multi-dimensional consistent strategy when using VisionReward as a reward model during preference optimization for visual generation. Experiments show that VisionReward can significantly outperform existing image and video reward models on both machine metrics and human evaluation. Notably, VisionReward surpasses VideoScore by 17.2% in preference prediction accuracy, and text-to-video models with VisionReward achieve a 31.6% higher pairwise win rate compared to the same models using VideoScore. All code and datasets are provided at https://github.com/THUDM/VisionReward.