Diffusion Models with Adaptive Negative Sampling Without External Resources

TL;DR

This paper introduces ANSWER, a training-free method that enhances diffusion models by internally understanding negation, leading to more faithful image generation without needing explicit negative prompts.

Contribution

It proposes a novel adaptive negative sampling technique that leverages classifier-free guidance to improve prompt adherence without external resources.

Findings

ANSWER outperforms baselines on multiple benchmarks.

Humans prefer images generated with ANSWER 2x more.

The method is applicable to any model supporting classifier-free guidance.

Abstract

Diffusion models (DMs) have demonstrated an unparalleled ability to create diverse and high-fidelity images from text prompts. However, they are also well-known to vary substantially regarding both prompt adherence and quality. Negative prompting was introduced to improve prompt compliance by specifying what an image must not contain. Previous works have shown the existence of an ideal negative prompt that can maximize the odds of the positive prompt. In this work, we explore relations between negative prompting and classifier-free guidance (CFG) to develop a sampling procedure, {\it Adaptive Negative Sampling Without External Resources} (ANSWER), that accounts for both positive and negative conditions from a single prompt. This leverages the internal understanding of negation by the diffusion model to increase the odds of generating images faithful to the prompt. ANSWER is a training-free technique, applicable to any model that supports CFG, and allows for negative grounding of image concepts without an explicit negative prompts, which are lossy and incomplete. Experiments show that adding ANSWER to existing DMs outperforms the baselines on multiple benchmarks and is preferred by humans 2x more over the other methods.