Control and Realism: Best of Both Worlds in Layout-to-Image without Training

TL;DR

This paper introduces WinWinLay, a training-free method for layout-to-image generation that improves control precision and realism by addressing attention biases and out-of-distribution artifacts, outperforming existing methods.

Contribution

WinWinLay proposes a novel training-free approach with non-local attention and adaptive updates to enhance layout control and image realism in diffusion models.

Findings

Outperforms state-of-the-art in layout control accuracy

Achieves higher photorealism in generated images

Effectively reduces artifacts and localization errors

Abstract

Layout-to-Image generation aims to create complex scenes with precise control over the placement and arrangement of subjects. Existing works have demonstrated that pre-trained Text-to-Image diffusion models can achieve this goal without training on any specific data; however, they often face challenges with imprecise localization and unrealistic artifacts. Focusing on these drawbacks, we propose a novel training-free method, WinWinLay. At its core, WinWinLay presents two key strategies, Non-local Attention Energy Function and Adaptive Update, that collaboratively enhance control precision and realism. On one hand, we theoretically demonstrate that the commonly used attention energy function introduces inherent spatial distribution biases, hindering objects from being uniformly aligned with layout instructions. To overcome this issue, non-local attention prior is explored to redistribute attention scores, facilitating objects to better conform to the specified spatial conditions. On the other hand, we identify that the vanilla backpropagation update rule can cause deviations from the pre-trained domain, leading to out-of-distribution artifacts. We accordingly introduce a Langevin dynamics-based adaptive update scheme as a remedy that promotes in-domain updating while respecting layout constraints. Extensive experiments demonstrate that WinWinLay excels in controlling element placement and achieving photorealistic visual fidelity, outperforming the current state-of-the-art methods.