Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures

TL;DR

Cafca introduces a novel volumetric prior trained on synthetic data that enables high-quality, expressive 3D face reconstruction and novel view synthesis from as few as three real-world images, outperforming existing methods.

Contribution

The paper proposes a synthetic-data-trained volumetric prior that generalizes to real-world faces, enabling expressive face modeling from minimal input views with fine details.

Findings

Requires only three input images for fine-tuning.

Outperforms state-of-the-art in quality of novel view synthesis.

Successfully captures detailed expressions and features.

Abstract

Volumetric modeling and neural radiance field representations have revolutionized 3D face capture and photorealistic novel view synthesis. However, these methods often require hundreds of multi-view input images and are thus inapplicable to cases with less than a handful of inputs. We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling from as few as three input views captured in the wild. Our key insight is that an implicit prior trained on synthetic data alone can generalize to extremely challenging real-world identities and expressions and render novel views with fine idiosyncratic details like wrinkles and eyelashes. We leverage a 3D Morphable Face Model to synthesize a large training set, rendering each identity with different expressions, hair, clothing, and other assets. We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject. On average, the fine-tuning requires only three inputs to cross the synthetic-to-real domain gap. The resulting personalized 3D model reconstructs strong idiosyncratic facial expressions and outperforms the state-of-the-art in high-quality novel view synthesis of faces from sparse inputs in terms of perceptual and photo-metric quality.