Aligning Diffusion Models by Optimizing Human Utility

TL;DR

Diffusion-KTO is a new method for aligning text-to-image diffusion models by maximizing expected human utility using simple binary feedback, improving alignment without complex reward models.

Contribution

It introduces a straightforward alignment approach that leverages binary feedback signals, avoiding the need for costly preference data or reward model training.

Findings

Outperforms existing methods in human judgment and automatic metrics.

Requires only simple binary feedback signals for alignment.

Enhances the applicability of aligning diffusion models with human preferences.

Abstract

We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.