Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator

TL;DR

This paper introduces GeoD, a geometry-aware discriminator for 3D-aware GANs, which guides the generator to produce more accurate 3D shapes and enhances multi-view consistency in image synthesis.

Contribution

The paper proposes a novel geometry-aware discriminator that guides the generator in 3D-aware GANs to improve 3D shape accuracy and multi-view consistency.

Findings

GeoD significantly improves 3D shape quality over state-of-the-art methods.

The approach enhances multi-view consistency in generated images.

Experiments show robustness across various architectures and datasets.

Abstract

3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough. In other words, displacing the generative mechanism only offers the capability, but not the guarantee, of producing 3D-aware images, because the supervision of the generator primarily comes from the discriminator. To address this issue, we propose GeoD through learning a geometry-aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such a simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Moreover, our approach is registered as a general framework such that a more capable discriminator (i.e., with a third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency.