Be Decisive: Noise-Induced Layouts for Multi-Subject Generation

TL;DR

This paper introduces a noise-aligned layout prediction method for multi-subject text-to-image generation, improving alignment and stability without external layout conflicts by refining a predicted spatial layout during denoising.

Contribution

It proposes a novel noise-induced layout prediction approach that aligns with the prompt and refines during denoising, avoiding conflicts with external layouts and enhancing multi-subject generation.

Findings

Improved text-image alignment over existing methods

More stable multi-subject generation results

Preserves diversity of the model's original distribution

Abstract

Generating multiple distinct subjects remains a challenge for existing text-to-image diffusion models. Complex prompts often lead to subject leakage, causing inaccuracies in quantities, attributes, and visual features. Preventing leakage among subjects necessitates knowledge of each subject's spatial location. Recent methods provide these spatial locations via an external layout control. However, enforcing such a prescribed layout often conflicts with the innate layout dictated by the sampled initial noise, leading to misalignment with the model's prior. In this work, we introduce a new approach that predicts a spatial layout aligned with the prompt, derived from the initial noise, and refines it throughout the denoising process. By relying on this noise-induced layout, we avoid conflicts with externally imposed layouts and better preserve the model's prior. Our method employs a small neural network to predict and refine the evolving noise-induced layout at each denoising step, ensuring clear boundaries between subjects while maintaining consistency. Experimental results show that this noise-aligned strategy achieves improved text-image alignment and more stable multi-subject generation compared to existing layout-guided techniques, while preserving the rich diversity of the model's original distribution.