LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Models

TL;DR

LumiCtrl is a novel method for personalized lighting control in text-to-image models, enabling precise scene illuminant manipulation from a single image, improving visual aesthetics and scene coherence.

Contribution

LumiCtrl introduces a new illuminant prompt learning approach with physics-based augmentation, prompt disentanglement, and contextual light adaptation for enhanced lighting control.

Findings

LumiCtrl outperforms existing methods in illuminant fidelity and aesthetic quality.

Qualitative and quantitative results demonstrate improved scene coherence.

Human studies favor LumiCtrl-generated images.

Abstract

Text-to-image (T2I) models have demonstrated remarkable progress in creative image generation, yet they still lack precise control over scene illuminants which is a crucial factor for content designers to manipulate visual aesthetics of generated images. In this paper, we present an illuminant personalization method named LumiCtrl that learns illuminant prompt given single image of the object. LumiCtrl consists of three components: given an image of the object, our method apply (a) physics-based illuminant augmentation along with Planckian locus to create fine-tuning variants under standard illuminants; (b) Edge-Guided Prompt Disentanglement using frozen ControlNet to ensure prompts focus on illumination, not the structure; and (c) a Masked Reconstruction Loss that focuses learning on foreground object while allowing background to adapt contextually which enables what we call Contextual Light Adaptation. We qualitatively and quantitatively compare LumiCtrl against other T2I customization methods. The results show that LumiCtrl achieves significantly better illuminant fidelity, aesthetic quality, and scene coherence compared to existing baselines. A human preference study further confirms the strong user preference for LumiCtrl generations.