LightMover: Generative Light Movement with Color and Intensity Controls

TL;DR

LightMover is a novel framework that enables controllable, physically plausible light editing in images by predicting illumination changes through a sequence-to-sequence model in visual token space.

Contribution

It introduces a unified approach for spatial and appearance light controls, along with an adaptive token-pruning mechanism to improve efficiency and fidelity.

Findings

Achieves high PSNR in light editing tasks.

Maintains semantic consistency across different control settings.

Reduces control sequence length by 41% with minimal fidelity loss.

Abstract

We present LightMover, a framework for controllable light manipulation in single images that leverages video diffusion priors to produce physically plausible illumination changes without re-rendering the scene. We formulate light editing as a sequence-to-sequence prediction problem in visual token space: given an image and light-control tokens, the model adjusts light position, color, and intensity together with resulting reflections, shadows, and falloff from a single view. This unified treatment of spatial (movement) and appearance (color, intensity) controls improves both manipulation and illumination understanding. We further introduce an adaptive token-pruning mechanism that preserves spatially informative tokens while compactly encoding non-spatial attributes, reducing control sequence length by 41% while maintaining editing fidelity. To train our framework, we construct a scalable rendering pipeline that generates large numbers of image pairs across varied light positions, colors, and intensities while keeping the scene content consistent with the original image. LightMover enables precise, independent control over light position, color, and intensity, and achieves high PSNR and strong semantic consistency (DINO, CLIP) across different tasks.