Aligning Diffusion Models with Noise-Conditioned Perception

TL;DR

This paper introduces a perceptual objective in the U-Net embedding space for diffusion models, improving human preference alignment, training efficiency, and visual quality compared to traditional pixel or VAE space optimization.

Contribution

It proposes a novel perceptual optimization approach in the U-Net embedding space for diffusion models, enhancing preference alignment and reducing training costs.

Findings

Outperforms standard latent-space methods in quality and efficiency

Achieves over 60% preference and visual appeal on SDXL

Reduces computational cost significantly during training

Abstract

Recent advancements in human preference optimization, initially developed for Language Models (LMs), have shown promise for text-to-image Diffusion Models, enhancing prompt alignment, visual appeal, and user preference. Unlike LMs, Diffusion Models typically optimize in pixel or VAE space, which does not align well with human perception, leading to slower and less efficient training during the preference alignment stage. We propose using a perceptual objective in the U-Net embedding space of the diffusion model to address these issues. Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and supervised fine-tuning (SFT) within this embedding space. This method significantly outperforms standard latent-space implementations across various metrics, including quality and computational cost. For SDXL, our approach provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt following against original open-sourced SDXL-DPO on the PartiPrompts dataset, while significantly reducing compute. Our approach not only improves the efficiency and quality of human preference alignment for diffusion models but is also easily integrable with other optimization techniques. The training code and LoRA weights will be available here: https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1