Real-time 3D-aware Portrait Video Relighting

TL;DR

This paper introduces a real-time 3D-aware relighting method for talking face videos using Neural Radiance Fields, enabling high-quality, view- and light-adjustable video synthesis at nearly 33 fps.

Contribution

It presents the first real-time 3D-aware relighting approach for in-the-wild talking face videos based on NeRF, with fast dual-encoders and temporal consistency for smooth, photo-realistic results.

Findings

Runs at 32.98 fps on consumer hardware

Achieves state-of-the-art in reconstruction quality and lighting accuracy

Ensures temporal stability with reduced flickering

Abstract

Synthesizing realistic videos of talking faces under custom lighting conditions and viewing angles benefits various downstream applications like video conferencing. However, most existing relighting methods are either time-consuming or unable to adjust the viewpoints. In this paper, we present the first real-time 3D-aware method for relighting in-the-wild videos of talking faces based on Neural Radiance Fields (NeRF). Given an input portrait video, our method can synthesize talking faces under both novel views and novel lighting conditions with a photo-realistic and disentangled 3D representation. Specifically, we infer an albedo tri-plane, as well as a shading tri-plane based on a desired lighting condition for each video frame with fast dual-encoders. We also leverage a temporal consistency network to ensure smooth transitions and reduce flickering artifacts. Our method runs at 32.98 fps on consumer-level hardware and achieves state-of-the-art results in terms of reconstruction quality, lighting error, lighting instability, temporal consistency and inference speed. We demonstrate the effectiveness and interactivity of our method on various portrait videos with diverse lighting and viewing conditions.