Lighting in Motion: Spatiotemporal HDR Lighting Estimation

TL;DR

LiMo is a diffusion-based method for spatiotemporal HDR lighting estimation that combines geometric and diffusion priors to produce detailed, accurate lighting maps for indoor and outdoor scenes.

Contribution

The paper introduces LiMo, a novel diffusion-based approach that integrates geometric conditions and a large dataset to improve lighting estimation accuracy and detail.

Findings

LiMo achieves state-of-the-art spatial control and prediction accuracy.

The method effectively combines diffuse and mirror predictions into HDRI maps.

LiMo outperforms previous methods in indoor and outdoor scene lighting estimation.

Abstract

We present Lighting in Motion (LiMo), a diffusion-based approach to spatiotemporal lighting estimation. LiMo targets both realistic high-frequency detail prediction and accurate illuminance estimation. To account for both, we propose generating a set of mirrored and diffuse spheres at different exposures, based on their 3D positions in the input. Making use of diffusion priors, we fine-tune powerful existing diffusion models on a large-scale customized dataset of indoor and outdoor scenes, paired with spatiotemporal light probes. For accurate spatial conditioning, we demonstrate that depth alone is insufficient and we introduce a new geometric condition to provide the relative position of the scene to the target 3D position. Finally, we combine diffuse and mirror predictions at different exposures into a single HDRI map leveraging differentiable rendering. We thoroughly evaluate our method and design choices to establish LiMo as state-of-the-art for both spatial control and prediction accuracy.