Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priors

TL;DR

This paper introduces a novel method for estimating spatiotemporally consistent indoor lighting from videos using diffusion priors, enabling zero-shot generalization and superior performance in in-the-wild scenes.

Contribution

It presents a new approach combining diffusion priors and light field modeling with a pre-trained diffusion model for indoor lighting estimation.

Findings

Outperforms baseline methods in indoor lighting estimation.

Achieves spatiotemporal consistency in in-the-wild videos.

Demonstrates zero-shot generalization to diverse scenes.

Abstract

Indoor lighting estimation from a single image or video remains a challenge due to its highly ill-posed nature, especially when the lighting condition of the scene varies spatially and temporally. We propose a method that estimates from an input video a continuous light field describing the spatiotemporally varying lighting of the scene. We leverage 2D diffusion priors for optimizing such light field represented as a MLP. To enable zero-shot generalization to in-the-wild scenes, we fine-tune a pre-trained image diffusion model to predict lighting at multiple locations by jointly inpainting multiple chrome balls as light probes. We evaluate our method on indoor lighting estimation from a single image or video and show superior performance over compared baselines. Most importantly, we highlight results on spatiotemporally consistent lighting estimation from in-the-wild videos, which is rarely demonstrated in previous works.