Feedback Guidance of Diffusion Models

TL;DR

This paper introduces FeedBack Guidance (FBG), a dynamic guidance method for diffusion models that self-regulates guidance strength based on sample needs, improving sample quality and diversity over traditional fixed guidance methods.

Contribution

The paper presents a novel, mathematically grounded guidance scheme that adapts guidance strength during inference, outperforming standard methods like CFG on benchmark datasets.

Findings

FBG outperforms CFG on ImageNet benchmark

Automatically adjusts guidance based on prompt complexity

Can be combined with existing guidance schemes

Abstract

While Classifier-Free Guidance (CFG) has become standard for improving sample fidelity in conditional diffusion models, it can harm diversity and induce memorization by applying constant guidance regardless of whether a particular sample needs correction. We propose FeedBack Guidance (FBG), which uses a state-dependent coefficient to self-regulate guidance amounts based on need. Our approach is derived from first principles by assuming the learned conditional distribution is linearly corrupted by the unconditional distribution, contrasting with CFG's implicit multiplicative assumption. Our scheme relies on feedback of its own predictions about the conditional signal informativeness to adapt guidance dynamically during inference, challenging the view of guidance as a fixed hyperparameter. The approach is benchmarked on ImageNet512x512, where it significantly outperforms Classifier-Free Guidance and is competitive to Limited Interval Guidance (LIG) while benefitting from a strong mathematical framework. On Text-To-Image generation, we demonstrate that, as anticipated, our approach automatically applies higher guidance scales for complex prompts than for simpler ones and that it can be easily combined with existing guidance schemes such as CFG or LIG.