The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noise

TL;DR

This paper uncovers trigger patches in initial noise that influence object placement in diffusion model images, introduces a detection method for these patches, and proposes a reject-sampling strategy to enhance generation quality.

Contribution

It identifies universal trigger patches in initial noise, develops a detector for these patches, and proposes a reject-sampling method to improve object placement and prompt adherence in diffusion models.

Findings

Trigger patches are universal and can be transferred across images.

A dataset and detector for trigger patches are developed.

Reject-sampling improves object placement accuracy.

Abstract

Diffusion models have achieved remarkable success in text-to-image generation tasks; however, the role of initial noise has been rarely explored. In this study, we identify specific regions within the initial noise image, termed trigger patches, that play a key role for object generation in the resulting images. Notably, these patches are ``universal'' and can be generalized across various positions, seeds, and prompts. To be specific, extracting these patches from one noise and injecting them into another noise leads to object generation in targeted areas. We identify these patches by analyzing the dispersion of object bounding boxes across generated images, leading to the development of a posterior analysis technique. Furthermore, we create a dataset consisting of Gaussian noises labeled with bounding boxes corresponding to the objects appearing in the generated images and train a detector that identifies these patches from the initial noise. To explain the formation of these patches, we reveal that they are outliers in Gaussian noise, and follow distinct distributions through two-sample tests. Finally, we find the misalignment between prompts and the trigger patch patterns can result in unsuccessful image generations. The study proposes a reject-sampling strategy to obtain optimal noise, aiming to improve prompt adherence and positional diversity in image generation.