DreamLight: Towards Harmonious and Consistent Image Relighting

TL;DR

DreamLight is a universal image relighting model that seamlessly composites subjects into new backgrounds, maintaining aesthetic lighting and color consistency, and supports both image-based and text-based background specification.

Contribution

The paper introduces a novel unified framework with a Position-Guided Light Adapter and Spectral Foreground Fixer for realistic, natural relighting in both image and text-guided scenarios.

Findings

Achieves superior relighting quality in experiments.

Effectively handles both image-based and text-based background inputs.

User studies confirm high satisfaction with results.

Abstract

We introduce a model named DreamLight for universal image relighting in this work, which can seamlessly composite subjects into a new background while maintaining aesthetic uniformity in terms of lighting and color tone. The background can be specified by natural images (image-based relighting) or generated from unlimited text prompts (text-based relighting). Existing studies primarily focus on image-based relighting, while with scant exploration into text-based scenarios. Some works employ intricate disentanglement pipeline designs relying on environment maps to provide relevant information, which grapples with the expensive data cost required for intrinsic decomposition and light source. Other methods take this task as an image translation problem and perform pixel-level transformation with autoencoder architecture. While these methods have achieved decent harmonization effects, they struggle to generate realistic and natural light interaction effects between the foreground and background. To alleviate these challenges, we reorganize the input data into a unified format and leverage the semantic prior provided by the pretrained diffusion model to facilitate the generation of natural results. Moreover, we propose a Position-Guided Light Adapter (PGLA) that condenses light information from different directions in the background into designed light query embeddings, and modulates the foreground with direction-biased masked attention. In addition, we present a post-processing module named Spectral Foreground Fixer (SFF) to adaptively reorganize different frequency components of subject and relighted background, which helps enhance the consistency of foreground appearance. Extensive comparisons and user study demonstrate that our DreamLight achieves remarkable relighting performance.