Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models

TL;DR

This paper introduces Diffusion-NPO, a method that trains a model to recognize negative preferences, improving the alignment of diffusion model outputs with human preferences without extensive retraining.

Contribution

It proposes a simple, effective approach to handle negative preferences in diffusion models, enhancing preference alignment without new datasets or training strategies.

Findings

Improves alignment of diffusion models with human preferences.

Seamlessly integrates with existing diffusion models like SD1.5 and SDXL.

Enhances avoidance of undesirable outputs.

Abstract

Diffusion models have made substantial advances in image generation, yet models trained on large, unfiltered datasets often yield outputs misaligned with human preferences. Numerous methods have been proposed to fine-tune pre-trained diffusion models, achieving notable improvements in aligning generated outputs with human preferences. However, we argue that existing preference alignment methods neglect the critical role of handling unconditional/negative-conditional outputs, leading to a diminished capacity to avoid generating undesirable outcomes. This oversight limits the efficacy of classifier-free guidance~(CFG), which relies on the contrast between conditional generation and unconditional/negative-conditional generation to optimize output quality. In response, we propose a straightforward but versatile effective approach that involves training a model specifically attuned to negative preferences. This method does not require new training strategies or datasets but rather involves minor modifications to existing techniques. Our approach integrates seamlessly with models such as SD1.5, SDXL, video diffusion models and models that have undergone preference optimization, consistently enhancing their alignment with human preferences.