GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

TL;DR

GRAF introduces a generative model based on radiance fields that enables high-resolution, 3D-aware image synthesis with disentangled camera and scene properties, trained solely on unposed 2D images.

Contribution

The paper presents a novel generative radiance field model that achieves high-resolution, 3D-consistent image synthesis with disentangled scene and camera attributes, surpassing voxel-based methods.

Findings

High-resolution 3D-aware image synthesis achieved

Disentanglement of camera and scene properties demonstrated

Model trained effectively on unposed 2D images

Abstract

While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxel-based representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.