Localized Control in Diffusion Models via Latent Vector Prediction

TL;DR

This paper presents a novel method for achieving precise local control in diffusion-based image generation, allowing user-defined regional modifications while maintaining overall prompt fidelity.

Contribution

It introduces a new training framework that uses masking and latent vector prediction to enable localized control in diffusion models, improving upon uniform conditioning methods.

Findings

Effective local control over image regions demonstrated

High-quality image synthesis with regional modifications achieved

Outperforms existing methods in localized editing accuracy

Abstract

Diffusion models emerged as a leading approach in text-to-image generation, producing high-quality images from textual descriptions. However, attempting to achieve detailed control to get a desired image solely through text remains a laborious trial-and-error endeavor. Recent methods have introduced image-level controls alongside with text prompts, using prior images to extract conditional information such as edges, segmentation and depth maps. While effective, these methods apply conditions uniformly across the entire image, limiting localized control. In this paper, we propose a novel methodology to enable precise local control over user-defined regions of an image, while leaving to the diffusion model the task of autonomously generating the remaining areas according to the original prompt. Our approach introduces a new training framework that incorporates masking features and an additional loss term, which leverages the prediction of the initial latent vector at any diffusion step to enhance the correspondence between the current step and the final sample in the latent space. Extensive experiments demonstrate that our method effectively synthesizes high-quality images with controlled local conditions.