GVP: Generative Volumetric Primitives

TL;DR

GVP introduces a novel pure 3D generative model capable of real-time high-resolution image synthesis by modeling volumetric primitives, maintaining multiview consistency and efficiency.

Contribution

It is the first pure 3D generator that efficiently produces high-resolution images by modeling volumetric primitives with a novel training technique.

Findings

Achieves real-time 512-resolution image synthesis.

Demonstrates superior efficiency over state-of-the-art models.

Maintains high multiview consistency.

Abstract

Advances in 3D-aware generative models have pushed the boundary of image synthesis with explicit camera control. To achieve high-resolution image synthesis, several attempts have been made to design efficient generators, such as hybrid architectures with both 3D and 2D components. However, such a design compromises multiview consistency, and the design of a pure 3D generator with high resolution is still an open problem. In this work, we present Generative Volumetric Primitives (GVP), the first pure 3D generative model that can sample and render 512-resolution images in real-time. GVP jointly models a number of volumetric primitives and their spatial information, both of which can be efficiently generated via a 2D convolutional network. The mixture of these primitives naturally captures the sparsity and correspondence in the 3D volume. The training of such a generator with a high degree of freedom is made possible through a knowledge distillation technique. Experiments on several datasets demonstrate superior efficiency and 3D consistency of GVP over the state-of-the-art.