Preference Alignment for Diffusion Model via Explicit Denoised Distribution Estimation

TL;DR

This paper introduces Denoised Distribution Estimation (DDE) for diffusion models, enabling preference alignment across entire denoising trajectories by explicitly connecting intermediate steps to terminal distributions, improving generation quality.

Contribution

It proposes two novel estimation strategies for DDE, allowing effective preference alignment throughout the denoising process, which was challenging in prior methods.

Findings

DDE improves preference alignment in diffusion models.

The approach outperforms existing methods quantitatively.

It enhances qualitative quality of generated images.

Abstract

Diffusion models have shown remarkable success in text-to-image generation, making preference alignment for these models increasingly important. The preference labels are typically available only at the terminal of denoising trajectories, which poses challenges in optimizing the intermediate denoising steps. In this paper, we propose to conduct Denoised Distribution Estimation (DDE) that explicitly connects intermediate steps to the terminal denoised distribution. Therefore, preference labels can be used for the entire trajectory optimization. To this end, we design two estimation strategies for our DDE. The first is stepwise estimation, which utilizes the conditional denoised distribution to estimate the model denoised distribution. The second is single-shot estimation, which converts the model output into the terminal denoised distribution via DDIM modeling. Analytically and empirically, we reveal that DDE equipped with two estimation strategies naturally derives a novel credit assignment scheme that prioritizes optimizing the middle part of the denoising trajectory. Extensive experiments demonstrate that our approach achieves superior performance, both quantitatively and qualitatively.