Rethinking Direct Preference Optimization in Diffusion Models

TL;DR

This paper introduces a novel approach to improve preference optimization in diffusion models by using a stable reference update strategy and timestep-aware training, leading to better alignment with human preferences.

Contribution

It proposes a new reference model update and training strategy that enhance exploration and address reward imbalance in diffusion preference optimization.

Findings

Improved performance on human preference benchmarks.

Enhanced exploration stability during training.

Compatibility with various preference optimization algorithms.

Abstract

Aligning text-to-image (T2I) diffusion models with human preferences has emerged as a critical research challenge. While recent advances in this area have extended preference optimization techniques from large language models (LLMs) to the diffusion setting, they often struggle with limited exploration. In this work, we propose a novel and orthogonal approach to enhancing diffusion-based preference optimization. First, we introduce a stable reference model update strategy that relaxes the frozen reference model, encouraging exploration while maintaining a stable optimization anchor through reference model regularization. Second, we present a timestep-aware training strategy that mitigates the reward scale imbalance problem across timesteps. Our method can be integrated into various preference optimization algorithms. Experimental results show that our approach improves the performance of state-of-the-art methods on human preference evaluation benchmarks. The code is available at the Github: https://github.com/kaist-cvml/RethinkingDPO_Diffusion_Models.