Relit-LiVE: Relight Video by Jointly Learning Environment Video

TL;DR

Relit-LiVE introduces a novel video relighting framework that achieves physically consistent, temporally stable results without prior camera pose knowledge by jointly predicting relit videos and environment maps.

Contribution

The method explicitly incorporates raw reference images and proposes a joint environment video prediction formulation to improve physical consistency and handle dynamic lighting and camera motion.

Findings

Outperforms state-of-the-art methods on synthetic and real-world benchmarks.

Supports dynamic lighting and camera motion with improved physical consistency.

Enables downstream applications like scene rendering and object insertion.

Abstract

Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric-illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE.