BodyReLux: Temporally Consistent Full-Body Video Relighting

TL;DR

BodyReLux is a novel video diffusion framework that enables temporally consistent, photorealistic relighting of full-body human performances, leveraging a hybrid dataset and advanced conditioning techniques.

Contribution

It introduces a subject-specific, temporally consistent video relighting method trained on a hybrid dataset with a new lighting conditioning approach.

Findings

Achieves high-quality, photorealistic relighting results.

Ensures temporal consistency in relighted videos.

Supports dynamic lighting control with masked attention.

Abstract

Being able to relight human performance is a fundamental task for post production and content creation. We present BodyReLux, a subject-specific video diffusion-based framework for relighting full-body human performances in a temporally consistent way. Our model is trained on a hybrid dataset of pixel-aligned video relighting pairs, covering a diverse combination of lighting conditions, performances and viewpoints. To acquire such dataset, we combine traditional static One-Light-at-a-Time (OLAT) capture and a novel dynamic performance capture in which two smoothly varying lighting sequences are rapidly interleaved. Because the lighting operates above the human flicker-fusion threshold, the interleaving does not appear to strobe. We train our video relighting model from a pretrained text-to-video model to fully leverage the generative priors for producing high quality videos. To achieve accurate lighting control, we introduce a new lighting conditioning method that represents each light source as a token. We further condition on sequences of lighting using masked attention to support dynamic lighting control. Together with a carefully designed data augmentation pipeline, we achieve photorealistic, robust, and temporally consistent video relighting of subject-specific human performances.