PC-Diffusion: Aligning Diffusion Models with Human Preferences via Preference Classifier

TL;DR

PC-Diffusion introduces a lightweight preference classifier to align diffusion models with human preferences, reducing training costs and improving stability without fine-tuning the entire model or relying on reference models.

Contribution

It proposes a novel, decoupled preference alignment framework using a trainable classifier, addressing limitations of existing DPO methods in diffusion models.

Findings

Achieves preference alignment comparable to DPO

Reduces training costs significantly

Enables stable, preference-guided generation

Abstract

Diffusion models have achieved remarkable success in conditional image generation, yet their outputs often remain misaligned with human preferences. To address this, recent work has applied Direct Preference Optimization (DPO) to diffusion models, yielding significant improvements.~However, DPO-like methods exhibit two key limitations: 1) High computational cost,due to the entire model fine-tuning; 2) Sensitivity to reference model quality}, due to its tendency to introduce instability and bias. To overcome these limitations, we propose a novel framework for human preference alignment in diffusion models (PC-Diffusion), using a lightweight, trainable Preference Classifier that directly models the relative preference between samples. By restricting preference learning to this classifier, PC-Diffusion decouples preference alignment from the generative model, eliminating the need for entire model fine-tuning and reference model reliance.~We further provide theoretical guarantees for PC-Diffusion:1) PC-Diffusion ensures that the preference-guided distributions are consistently propagated across timesteps. 2)The training objective of the preference classifier is equivalent to DPO, but does not require a reference model.3) The proposed preference-guided correction can progressively steer generation toward preference-aligned regions.~Empirical results show that PC-Diffusion achieves comparable preference consistency to DPO while significantly reducing training costs and enabling efficient and stable preference-guided generation.