Understanding the Impact of Negative Prompts: When and How Do They Take Effect?

TL;DR

This paper investigates how negative prompts influence image generation in models like Stable Diffusion, revealing two main behaviors—delayed effects and mutual cancellation—and demonstrating their practical applications such as object inpainting.

Contribution

First comprehensive analysis of negative prompts' mechanisms, identifying delayed effects and mutual cancellation, with practical methods for improved image editing.

Findings

Negative prompts cause delayed effects in image generation.

Negative prompts can delete concepts via mutual cancellation in latent space.

Negative prompts enable effective object inpainting with minimal background changes.

Abstract

The concept of negative prompts, emerging from conditional generation models like Stable Diffusion, allows users to specify what to exclude from the generated images.%, demonstrating significant practical efficacy. Despite the widespread use of negative prompts, their intrinsic mechanisms remain largely unexplored. This paper presents the first comprehensive study to uncover how and when negative prompts take effect. Our extensive empirical analysis identifies two primary behaviors of negative prompts. Delayed Effect: The impact of negative prompts is observed after positive prompts render corresponding content. Deletion Through Neutralization: Negative prompts delete concepts from the generated image through a mutual cancellation effect in latent space with positive prompts. These insights reveal significant potential real-world applications; for example, we demonstrate that negative prompts can facilitate object inpainting with minimal alterations to the background via a simple adaptive algorithm. We believe our findings will offer valuable insights for the community in capitalizing on the potential of negative prompts.