A New Dimension in Testimony: Relighting Video with Reflectance Field Exemplars
Loc Huynh, Bipin Kishore, Paul Debevec

TL;DR
This paper introduces a learning-based approach to estimate a 4D reflectance field from video footage, enabling realistic relighting of subjects under new lighting conditions with improved accuracy and efficiency.
Contribution
It presents a novel semi-supervised neural network framework that estimates reflectance fields from video, incorporating a differentiable renderer for enhanced relighting accuracy.
Findings
Outperforms state-of-the-art in realism and speed
Handles unseen poses effectively
Accurately estimates lighting and reflectance from video
Abstract
We present a learning-based method for estimating 4D reflectance field of a person given video footage illuminated under a flat-lit environment of the same subject. For training data, we use one light at a time to illuminate the subject and capture the reflectance field data in a variety of poses and viewpoints. We estimate the lighting environment of the input video footage and use the subject's reflectance field to create synthetic images of the subject illuminated by the input lighting environment. We then train a deep convolutional neural network to regress the reflectance field from the synthetic images. We also use a differentiable renderer to provide feedback for the network by matching the relit images with the input video frames. This semi-supervised training scheme allows the neural network to handle unseen poses in the dataset as well as compensate for the lighting estimation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Computer Graphics and Visualization Techniques · Generative Adversarial Networks and Image Synthesis
