Towards General Preference Alignment: Diffusion Models at Nash Equilibrium

TL;DR

This paper introduces Diffusion Nash Preference Optimization (Diff.-NPO), a game-theoretic approach for aligning diffusion models with human preferences, outperforming existing methods in text-to-image generation tasks.

Contribution

It proposes a novel game-theoretic framework for diffusion alignment that does not rely on reward models or the Bradley--Terry assumption, improving preference alignment.

Findings

Diff.-NPO outperforms existing preference-based methods in text-to-image tasks.

The framework encourages self-play to improve alignment.

Empirical results demonstrate better metrics compared to prior approaches.

Abstract

Reinforcement learning from human feedback (RLHF) has been popular for aligning text-to-image (T2I) diffusion models with human preferences. As a mainstream branch of RLHF, Direct Preference Optimization (DPO) offers a computationally efficient alternative that avoids explicit reward modeling and has been widely adopted in diffusion alignment. However, existing preference-based methods for diffusion alignment still rely on reward-induced preference signals and typically assume that human preferences can be adequately modeled by the Bradley--Terry (BT) model, which may fail to capture the full complexity of human preferences. In this paper, we formulate diffusion alignment from a game-theoretic perspective. We propose Diffusion Nash Preference Optimization (Diff.-NPO), an intuitive general preference framework for diffusion alignment. Diff.-NPO encourages the current policy to play against itself to achieve self improvement and lead to a better alignment. Empirically, we demonstrate the effectiveness of Diff.-NPO on the text-to-image generation task via various metrics. Diff.-NPO consistently outperforms existing preference-based diffusion alignment methods.