Linear-DPO: Linear Direct Preference Optimization for Diffusion and Flow-Matching Generative Models

TL;DR

This paper introduces Linear-DPO, a unified preference optimization method for diffusion and flow-matching models, improving text-to-image generation quality by addressing objective mismatches and replacing utility functions.

Contribution

It derives a generalized DPO objective applicable to both diffusion and flow-matching models and proposes Linear-DPO with a linear utility function and reference model, enhancing generation performance.

Findings

Linear-DPO outperforms existing baselines on SD1.5, SDXL, and SD3-Medium.

Unified framework for diffusion and flow-matching models.

Replacing sigmoid utility with linear utility improves results.

Abstract

Direct Preference Optimization (DPO) is successful for alignment in LLMs but still faces challenges in text-to-image generation. Existing studies are confined to denoising diffusion models while overlooking flow-matching, and suffer from an objective mismatch when applying discrete NLP-based DPO to regression-based generative tasks.\ In this paper, we derive a generalized DPO objective that covers both diffusion and flow-matching via a unified reverse-time SDE framework, and point out from a gradient perspective that the standard DPO objective is suboptimal for text-to-image generation. Consequently, we propose Linear-DPO, which replaces the aggressive sigmoid-based utility function with a sustained linear utility and incorporates an EMA-updated reference model. Qualitative and quantitative experiments on diffusion models (SD1.5, SDXL) and flow-matching model (SD3-Medium) demonstrate the superiority of our approach over existing baselines.