Dynamic Negative Guidance of Diffusion Models

TL;DR

This paper introduces Dynamic Negative Guidance (DNG), a novel method for diffusion models that adapt guidance based on time and state, improving safety and image quality over traditional negative prompting.

Contribution

The paper proposes a principled, state-dependent modulation technique called DNG for diffusion models, enhancing control without extra training or significant computational cost.

Findings

DNG improves safety and class balance in MNIST and CIFAR10.

DNG yields higher image quality compared to baseline methods.

DNG provides more accurate guidance than traditional negative prompting.

Abstract

Negative Prompting (NP) is widely utilized in diffusion models, particularly in text-to-image applications, to prevent the generation of undesired features. In this paper, we show that conventional NP is limited by the assumption of a constant guidance scale, which may lead to highly suboptimal results, or even complete failure, due to the non-stationarity and state-dependence of the reverse process. Based on this analysis, we derive a principled technique called Dynamic Negative Guidance, which relies on a near-optimal time and state dependent modulation of the guidance without requiring additional training. Unlike NP, negative guidance requires estimating the posterior class probability during the denoising process, which is achieved with limited additional computational overhead by tracking the discrete Markov Chain during the generative process. We evaluate the performance of DNG class-removal on MNIST and CIFAR10, where we show that DNG leads to higher safety, preservation of class balance and image quality when compared with baseline methods. Furthermore, we show that it is possible to use DNG with Stable Diffusion to obtain more accurate and less invasive guidance than NP.