EpiGRAF: Rethinking training of 3D GANs

TL;DR

EpiGRAF introduces a novel patch-wise training approach for high-resolution 3D-aware generative models, achieving state-of-the-art image quality and faster training times without relying on volumetric rendering or low-fidelity geometry.

Contribution

The paper presents a new patch-wise training method with a location- and scale-aware discriminator and an annealed beta sampling strategy, enabling high-resolution 3D GANs with improved efficiency and quality.

Findings

Achieves state-of-the-art image quality at 256^2 and 512^2 resolutions.

Trains approximately 2.5 times faster than upsampler-based methods.

Maintains high-fidelity geometry and multi-view consistency.

Abstract

A very recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely on volumetric rendering, which is expensive to employ at high resolutions. During the past months, there appeared more than 10 works that address this scaling issue by training a separate 2D decoder to upsample a low-resolution image (or a feature tensor) produced from a pure 3D generator. But this solution comes at a cost: not only does it break multi-view consistency (i.e. shape and texture change when the camera moves), but it also learns the geometry in a low fidelity. In this work, we show that it is possible to obtain a high-resolution 3D generator with SotA image quality by following a completely different route of simply training the model patch-wise. We revisit and improve this optimization scheme in two ways. First, we design a location- and scale-aware discriminator to work on patches of different proportions and spatial positions. Second, we modify the patch sampling strategy based on an annealed beta distribution to stabilize training and accelerate the convergence. The resulted model, named EpiGRAF, is an efficient, high-resolution, pure 3D generator, and we test it on four datasets (two introduced in this work) at $256^2$ and $512^2$ resolutions. It obtains state-of-the-art image quality, high-fidelity geometry and trains ${\approx} 2.5 \times$ faster than the upsampler-based counterparts. Project website: https://universome.github.io/epigraf.