SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedback

TL;DR

SAIL is a novel framework enabling diffusion models to self-improve and align with human preferences using minimal feedback, eliminating the need for large datasets or reward models, and outperforming existing methods.

Contribution

The paper introduces SAIL, a self-iterative learning approach allowing diffusion models to self-align with human preferences using minimal data without external reward models.

Findings

SAIL outperforms state-of-the-art methods with only 6% of the preference data.

Diffusion models can self-annotate and improve without large-scale human-labeled datasets.

The ranked preference mixup strategy enhances robust self-improvement.

Abstract

Aligning diffusion models with human preferences remains challenging, particularly when reward models are unavailable or impractical to obtain, and collecting large-scale preference datasets is prohibitively expensive. \textit{This raises a fundamental question: can we achieve effective alignment using only minimal human feedback, without auxiliary reward models, by unlocking the latent capabilities within diffusion models themselves?} In this paper, we propose \textbf{SAIL} (\textbf{S}elf-\textbf{A}mplified \textbf{I}terative \textbf{L}earning), a novel framework that enables diffusion models to act as their own teachers through iterative self-improvement. Starting from a minimal seed set of human-annotated preference pairs, SAIL operates in a closed-loop manner where the model progressively generates diverse samples, self-annotates preferences based on its evolving understanding, and refines itself using this self-augmented dataset. To ensure robust learning and prevent catastrophic forgetting, we introduce a ranked preference mixup strategy that carefully balances exploration with adherence to initial human priors. Extensive experiments demonstrate that SAIL consistently outperforms state-of-the-art methods across multiple benchmarks while using merely 6\% of the preference data required by existing approaches, revealing that diffusion models possess remarkable self-improvement capabilities that, when properly harnessed, can effectively replace both large-scale human annotation and external reward models.