LightCtrl: Training-free Controllable Video Relighting

TL;DR

LightCtrl is a training-free, controllable video relighting method that allows explicit user control over illumination through a specified light trajectory, combining pre-trained diffusion models with novel modules for coherence and adherence.

Contribution

It introduces a training-free approach for controllable video relighting that explicitly follows user-defined light trajectories using innovative modules for light injection and geometry-aware relighting.

Findings

Produces high-quality videos with diverse, trajectory-following illumination changes.

Outperforms baseline methods in controllability and visual coherence.

Code is publicly available at the provided GitHub URL.

Abstract

Recent diffusion models have achieved remarkable success in image relighting, and this success has quickly been extended to video relighting. However, existing methods offer limited explicit control over illumination in the relighted output. We present LightCtrl, the first controllable video relighting method that enables explicit control of video illumination through a user-supplied light trajectory in a training-free manner. Our approach combines pre-trained diffusion models: an image relighting model processes each frame individually, followed by a video diffusion prior to enhance temporal consistency. To achieve explicit control over dynamically varying lighting, we introduce two key components. First, a Light Map Injection module samples light trajectory-specific noise and injects it into the latent representation of the source video, improving illumination coherence with the conditional light trajectory. Second, a Geometry-Aware Relighting module dynamically combines RGB and normal map latents in the frequency domain to suppress the influence of the original lighting, further enhancing adherence to the input light trajectory. Experiments show that LightCtrl produces high-quality videos with diverse illumination changes that closely follow the specified light trajectory, demonstrating improved controllability over baseline methods. Code is available at: https://github.com/GVCLab/LightCtrl.