MFTF: Mask-free Training-free Object Level Layout Control Diffusion Model

TL;DR

MFTF introduces a novel diffusion-based method that enables precise object layout control in generated images without masks or retraining, supporting multi-object positioning and semantic editing.

Contribution

It presents a training-free, mask-free diffusion model that allows accurate object position adjustments during image generation, a significant advance over existing methods.

Findings

Supports both single and multi-object positional adjustments

Enables simultaneous layout control and semantic editing

Achieves precise object placement without additional masks or retraining

Abstract

Text-to-image generation models have revolutionized content creation, but diffusion-based vision-language models still face challenges in precisely controlling the shape, appearance, and positional placement of objects in generated images using text guidance alone. Existing global image editing models rely on additional masks or images as guidance to achieve layout control, often requiring retraining of the model. While local object-editing models allow modifications to object shapes, they lack the capability to control object positions. To address these limitations, we propose the Mask-free Training-free Object-Level Layout Control Diffusion Model (MFTF), which provides precise control over object positions without requiring additional masks or images. The MFTF model supports both single-object and multi-object positional adjustments, such as translation and rotation, while enabling simultaneous layout control and object semantic editing. The MFTF model employs a parallel denoising process for both the source and target diffusion models. During this process, attention masks are dynamically generated from the cross-attention layers of the source diffusion model and applied to queries from the self-attention layers to isolate objects. These queries, generated in the source diffusion model, are then adjusted according to the layout control parameters and re-injected into the self-attention layers of the target diffusion model. This approach ensures accurate and precise positional control of objects. Project source code available at https://github.com/syang-genai/MFTF.