RelightVid: Temporal-Consistent Diffusion Model for Video Relighting

TL;DR

RelightVid is a novel framework that enables high-quality, temporally consistent video relighting using diffusion models, trained on diverse in-the-wild videos and capable of accepting various relighting conditions.

Contribution

It introduces a flexible, training-efficient diffusion-based method for video relighting that maintains temporal consistency without requiring intrinsic decomposition.

Findings

Achieves high temporal consistency in video relighting.

Operates with various relighting conditions like text prompts and environment maps.

Does not rely on intrinsic decomposition, preserving illumination priors.

Abstract

Diffusion models have demonstrated remarkable success in image generation and editing, with recent advancements enabling albedo-preserving image relighting. However, applying these models to video relighting remains challenging due to the lack of paired video relighting datasets and the high demands for output fidelity and temporal consistency, further complicated by the inherent randomness of diffusion models. To address these challenges, we introduce RelightVid, a flexible framework for video relighting that can accept background video, text prompts, or environment maps as relighting conditions. Trained on in-the-wild videos with carefully designed illumination augmentations and rendered videos under extreme dynamic lighting, RelightVid achieves arbitrary video relighting with high temporal consistency without intrinsic decomposition while preserving the illumination priors of its image backbone.