Seeing What Matters: Visual Preference Policy Optimization for Visual Generation

TL;DR

ViPO enhances visual generative models by incorporating pixel-level, spatially, and temporally aware feedback into reinforcement learning, leading to better alignment with human preferences and improved generalization.

Contribution

Introduces ViPO, a structured advantage method that leverages pretrained vision backbones to improve reinforcement learning for visual content generation.

Findings

ViPO outperforms vanilla GRPO on image and video benchmarks.

It improves in-domain alignment with human preferences.

It enhances out-of-domain generalization.

Abstract

Reinforcement learning (RL) has become a powerful tool for post-training visual generative models, with Group Relative Policy Optimization (GRPO) increasingly used to align generators with human preferences. However, existing GRPO pipelines rely on a single scalar reward per sample, treating each image or video as a holistic entity and ignoring the rich spatial and temporal structure of visual content. This coarse supervision hinders the correction of localized artifacts and the modeling of fine-grained perceptual cues. We introduce Visual Preference Policy Optimization (ViPO), a GRPO variant that lifts scalar feedback into structured, pixel-level advantages. ViPO employs a Perceptual Structuring Module that uses pretrained vision backbones to construct spatially and temporally aware advantage maps, redistributing optimization pressure toward perceptually important regions while preserving the stability of standard GRPO. Across both image and video benchmarks, ViPO consistently outperforms vanilla GRPO, improving in-domain alignment with human-preference rewards and enhancing generalization on out-of-domain evaluations. The method is architecture-agnostic, lightweight, and fully compatible with existing GRPO training pipelines, providing a more expressive and informative learning signal for visual generation.