A New Dimension in Testimony: Relighting Video with Reflectance Field Exemplars

TL;DR

This paper introduces a learning-based approach to estimate a 4D reflectance field from video footage, enabling realistic relighting of subjects under new lighting conditions with improved accuracy and efficiency.

Contribution

It presents a novel semi-supervised neural network framework that estimates reflectance fields from video, incorporating a differentiable renderer for enhanced relighting accuracy.

Findings

Outperforms state-of-the-art in realism and speed

Handles unseen poses effectively

Accurately estimates lighting and reflectance from video

Abstract

We present a learning-based method for estimating 4D reflectance field of a person given video footage illuminated under a flat-lit environment of the same subject. For training data, we use one light at a time to illuminate the subject and capture the reflectance field data in a variety of poses and viewpoints. We estimate the lighting environment of the input video footage and use the subject's reflectance field to create synthetic images of the subject illuminated by the input lighting environment. We then train a deep convolutional neural network to regress the reflectance field from the synthetic images. We also use a differentiable renderer to provide feedback for the network by matching the relit images with the input video frames. This semi-supervised training scheme allows the neural network to handle unseen poses in the dataset as well as compensate for the lighting estimation error. We evaluate our method on the video footage of the real Holocaust survivors and show that our method outperforms the state-of-the-art methods in both realism and speed.