Preface: A Data-driven Volumetric Prior for Few-shot Ultra High-resolution Face Synthesis

TL;DR

This paper introduces a data-driven volumetric prior for ultra high-resolution face synthesis that enables realistic, multi-view face generation from minimal input views, reducing hardware needs and broadening applicability.

Contribution

It presents a novel identity-conditioned NeRF model trained on low-res multi-view images, allowing high-quality face synthesis from just two casual images.

Findings

High-quality face synthesis from 2-3 views.

Model generalizes to unseen identities.

Requires minimal multi-view input at inference.

Abstract

NeRFs have enabled highly realistic synthesis of human faces including complex appearance and reflectance effects of hair and skin. These methods typically require a large number of multi-view input images, making the process hardware intensive and cumbersome, limiting applicability to unconstrained settings. We propose a novel volumetric human face prior that enables the synthesis of ultra high-resolution novel views of subjects that are not part of the prior's training distribution. This prior model consists of an identity-conditioned NeRF, trained on a dataset of low-resolution multi-view images of diverse humans with known camera calibration. A simple sparse landmark-based 3D alignment of the training dataset allows our model to learn a smooth latent space of geometry and appearance despite a limited number of training identities. A high-quality volumetric representation of a novel subject can be obtained by model fitting to 2 or 3 camera views of arbitrary resolution. Importantly, our method requires as few as two views of casually captured images as input at inference time.