Improving image synthesis with diffusion-negative sampling

TL;DR

This paper introduces diffusion-negative prompting (DNP), a novel method that automatically generates effective negative prompts for diffusion models by sampling least compliant images and translating them into negative prompts, improving image synthesis quality.

Contribution

The paper proposes diffusion-negative sampling (DNS), a simple, training-free technique to generate negative prompts that better align with human intuition, enhancing diffusion model performance.

Findings

DNP improves image quality and prompt adherence.

DNS is easy to implement and compatible with various diffusion models.

Human evaluations favor DNP-generated prompts.

Abstract

For image generation with diffusion models (DMs), a negative prompt n can be used to complement the text prompt p, helping define properties not desired in the synthesized image. While this improves prompt adherence and image quality, finding good negative prompts is challenging. We argue that this is due to a semantic gap between humans and DMs, which makes good negative prompts for DMs appear unintuitive to humans. To bridge this gap, we propose a new diffusion-negative prompting (DNP) strategy. DNP is based on a new procedure to sample images that are least compliant with p under the distribution of the DM, denoted as diffusion-negative sampling (DNS). Given p, one such image is sampled, which is then translated into natural language by the user or a captioning model, to produce the negative prompt n*. The pair (p, n*) is finally used to prompt the DM. DNS is straightforward to implement and requires no training. Experiments and human evaluations show that DNP performs well both quantitatively and qualitatively and can be easily combined with several DM variants.