Coherent 3D Portrait Video Reconstruction via Triplane Fusion

TL;DR

This paper introduces a fusion-based approach for 3D portrait video reconstruction that combines a personalized 3D prior with per-frame data, achieving realistic, temporally consistent 3D videos from a single image.

Contribution

It presents a novel fusion method that maintains both identity coherence and dynamic appearance, trained solely on synthetic data for improved accuracy and consistency.

Findings

Achieves state-of-the-art 3D reconstruction accuracy.

Ensures temporal stability in 3D portrait videos.

Works effectively on both studio and wild datasets.

Abstract

Recent breakthroughs in single-image 3D portrait reconstruction have enabled telepresence systems to stream 3D portrait videos from a single camera in real-time, potentially democratizing telepresence. However, per-frame 3D reconstruction exhibits temporal inconsistency and forgets the user's appearance. On the other hand, self-reenactment methods can render coherent 3D portraits by driving a personalized 3D prior, but fail to faithfully reconstruct the user's per-frame appearance (e.g., facial expressions and lighting). In this work, we recognize the need to maintain both coherent identity and dynamic per-frame appearance to enable the best possible realism. To this end, we propose a new fusion-based method that fuses a personalized 3D subject prior with per-frame information, producing temporally stable 3D videos with faithful reconstruction of the user's per-frame appearances. Trained only using synthetic data produced by an expression-conditioned 3D GAN, our encoder-based method achieves both state-of-the-art 3D reconstruction accuracy and temporal consistency on in-studio and in-the-wild datasets.