DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

TL;DR

DenseDPO enhances text-to-video diffusion models by enabling fine-grained, motion-neutral preference learning through aligned video pairs and segment-level annotations, significantly improving motion generation quality.

Contribution

It introduces DenseDPO, a novel method that creates aligned video pairs and uses segment-level preference labels, reducing data needs and bias in preference optimization for video diffusion models.

Findings

DenseDPO improves motion generation over vanilla DPO.

DenseDPO matches vanilla DPO in text alignment and visual quality.

Automatic preference annotation with VLMs is effective.

Abstract

Direct Preference Optimization (DPO) has recently been applied as a post-training technique for text-to-video diffusion models. To obtain training data, annotators are asked to provide preferences between two videos generated from independent noise. However, this approach prohibits fine-grained comparisons, and we point out that it biases the annotators towards low-motion clips as they often contain fewer visual artifacts. In this work, we introduce DenseDPO, a method that addresses these shortcomings by making three contributions. First, we create each video pair for DPO by denoising corrupted copies of a ground truth video. This results in aligned pairs with similar motion structures while differing in local details, effectively neutralizing the motion bias. Second, we leverage the resulting temporal alignment to label preferences on short segments rather than entire clips, yielding a denser and more precise learning signal. With only one-third of the labeled data, DenseDPO greatly improves motion generation over vanilla DPO, while matching it in text alignment, visual quality, and temporal consistency. Finally, we show that DenseDPO unlocks automatic preference annotation using off-the-shelf Vision Language Models (VLMs): GPT accurately predicts segment-level preferences similar to task-specifically fine-tuned video reward models, and DenseDPO trained on these labels achieves performance close to using human labels.