LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation

TL;DR

LGTM introduces a training-free method to control lighting in text-to-image diffusion models by manipulating initial noise, enabling fine-grained, user-guided lighting adjustments without fine-tuning or additional training.

Contribution

The paper presents a novel approach that manipulates initial noise in diffusion models for lighting control, eliminating the need for fine-tuning or retraining.

Findings

Outperforms prompt-based baselines in lighting consistency

Preserves image quality and text alignment

Works seamlessly with models like ControlNet

Abstract

Diffusion models have demonstrated high-quality performance in conditional text-to-image generation, particularly with structural cues such as edges, layouts, and depth. However, lighting conditions have received limited attention and remain difficult to control within the generative process. Existing methods handle lighting through a two-stage pipeline that relights images after generation, which is inefficient. Moreover, they rely on fine-tuning with large datasets and heavy computation, limiting their adaptability to new models and tasks. To address this, we propose a novel Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulation (LGTM), which manipulates the initial latent noise of the diffusion process to guide image generation with text prompts and user-specified light directions. Through a channel-wise analysis of the latent space, we find that selectively manipulating latent channels enables fine-grained lighting control without fine-tuning or modifying the pre-trained model. Extensive experiments show that our method surpasses prompt-based baselines in lighting consistency, while preserving image quality and text alignment. This approach introduces new possibilities for dynamic, user-guided light control. Furthermore, it integrates seamlessly with models like ControlNet, demonstrating adaptability across diverse scenarios.