Learning Detailed Radiance Manifolds for High-Fidelity and 3D-Consistent Portrait Synthesis from Monocular Image

TL;DR

This paper introduces a novel method for high-fidelity, 3D-consistent portrait synthesis from monocular images by enhancing radiance manifolds with detailed, learned features, outperforming previous approaches.

Contribution

It proposes a detail manifolds reconstructor that learns fine 3D-consistent details from monocular images, improving over coarse reconstructions and ensuring 3D consistency.

Findings

Achieves high-fidelity portrait synthesis with 3D consistency.

Outperforms prior art in multiview portrait generation.

Uses 3D priors to regulate learned details for realistic results.

Abstract

A key challenge for novel view synthesis of monocular portrait images is 3D consistency under continuous pose variations. Most existing methods rely on 2D generative models which often leads to obvious 3D inconsistency artifacts. We present a 3D-consistent novel view synthesis approach for monocular portrait images based on a recent proposed 3D-aware GAN, namely Generative Radiance Manifolds (GRAM), which has shown strong 3D consistency at multiview image generation of virtual subjects via the radiance manifolds representation. However, simply learning an encoder to map a real image into the latent space of GRAM can only reconstruct coarse radiance manifolds without faithful fine details, while improving the reconstruction fidelity via instance-specific optimization is time-consuming. We introduce a novel detail manifolds reconstructor to learn 3D-consistent fine details on the radiance manifolds from monocular images, and combine them with the coarse radiance manifolds for high-fidelity reconstruction. The 3D priors derived from the coarse radiance manifolds are used to regulate the learned details to ensure reasonable synthesized results at novel views. Trained on in-the-wild 2D images, our method achieves high-fidelity and 3D-consistent portrait synthesis largely outperforming the prior art.