G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images

TL;DR

G-NeRF enhances single-view novel view synthesis by leveraging 3D GANs for geometry priors and depth-aware training, enabling high-quality multi-view synthesis without multi-view training data.

Contribution

The paper introduces a novel method combining 3D GANs and depth-aware training to improve geometry priors from single-view images for view synthesis.

Findings

Effective multi-view synthesis from single-view images.

Improved geometry quality via truncation in 3D GANs.

Quantitative and qualitative results demonstrate superiority.

Abstract

Novel view synthesis aims to generate new view images of a given view image collection. Recent attempts address this problem relying on 3D geometry priors (e.g., shapes, sizes, and positions) learned from multi-view images. However, such methods encounter the following limitations: 1) they require a set of multi-view images as training data for a specific scene (e.g., face, car or chair), which is often unavailable in many real-world scenarios; 2) they fail to extract the geometry priors from single-view images due to the lack of multi-view supervision. In this paper, we propose a Geometry-enhanced NeRF (G-NeRF), which seeks to enhance the geometry priors by a geometry-guided multi-view synthesis approach, followed by a depth-aware training. In the synthesis process, inspired that existing 3D GAN models can unconditionally synthesize high-fidelity multi-view images, we seek to adopt off-the-shelf 3D GAN models, such as EG3D, as a free source to provide geometry priors through synthesizing multi-view data. Simultaneously, to further improve the geometry quality of the synthetic data, we introduce a truncation method to effectively sample latent codes within 3D GAN models. To tackle the absence of multi-view supervision for single-view images, we design the depth-aware training approach, incorporating a depth-aware discriminator to guide geometry priors through depth maps. Experiments demonstrate the effectiveness of our method in terms of both qualitative and quantitative results.