SpotActor: Training-Free Layout-Controlled Consistent Image Generation

TL;DR

SpotActor introduces a training-free, layout-controlled image generation method that ensures consistent placement and appearance of subjects across images, advancing artistic creation and comic production.

Contribution

It pioneers the Layout-to-Consistent-Image generation task with a novel dual energy guidance formalization and a training-free pipeline, including innovative attention mechanisms.

Findings

Achieves superior layout alignment and subject consistency.

Demonstrates effectiveness through comprehensive experiments.

Outperforms existing methods in prompt conformity and background diversity.

Abstract

Text-to-image diffusion models significantly enhance the efficiency of artistic creation with high-fidelity image generation. However, in typical application scenarios like comic book production, they can neither place each subject into its expected spot nor maintain the consistent appearance of each subject across images. For these issues, we pioneer a novel task, Layout-to-Consistent-Image (L2CI) generation, which produces consistent and compositional images in accordance with the given layout conditions and text prompts. To accomplish this challenging task, we present a new formalization of dual energy guidance with optimization in a dual semantic-latent space and thus propose a training-free pipeline, SpotActor, which features a layout-conditioned backward update stage and a consistent forward sampling stage. In the backward stage, we innovate a nuanced layout energy function to mimic the attention activations with a sigmoid-like objective. While in the forward stage, we design Regional Interconnection Self-Attention (RISA) and Semantic Fusion Cross-Attention (SFCA) mechanisms that allow mutual interactions across images. To evaluate the performance, we present ActorBench, a specified benchmark with hundreds of reasonable prompt-box pairs stemming from object detection datasets. Comprehensive experiments are conducted to demonstrate the effectiveness of our method. The results prove that SpotActor fulfills the expectations of this task and showcases the potential for practical applications with superior layout alignment, subject consistency, prompt conformity and background diversity.