Generative Multiplane Images: Making a 2D GAN 3D-Aware

TL;DR

This paper introduces a simple yet effective modification to StyleGANv2 to create 3D-aware images called GMPIs, enabling view-consistent high-quality renderings with flexible memory use and fast training at high resolution.

Contribution

The paper demonstrates that only two modifications are needed to make a 2D GAN 3D-aware, resulting in view-consistent, high-quality images with flexible alpha map adjustments.

Findings

GMPIs produce view-consistent high-quality images.

Training GMPIs is fast, taking less than half a day at 1024^2 resolution.

The method generalizes well across multiple high-resolution datasets.

Abstract

What is really needed to make an existing 2D GAN 3D-aware? To answer this question, we modify a classical GAN, i.e., StyleGANv2, as little as possible. We find that only two modifications are absolutely necessary: 1) a multiplane image style generator branch which produces a set of alpha maps conditioned on their depth; 2) a pose-conditioned discriminator. We refer to the generated output as a 'generative multiplane image' (GMPI) and emphasize that its renderings are not only high-quality but also guaranteed to be view-consistent, which makes GMPIs different from many prior works. Importantly, the number of alpha maps can be dynamically adjusted and can differ between training and inference, alleviating memory concerns and enabling fast training of GMPIs in less than half a day at a resolution of $1024^2$. Our findings are consistent across three challenging and common high-resolution datasets, including FFHQ, AFHQv2, and MetFaces.