Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation

TL;DR

Mimic3D introduces a novel 3D-to-2D imitation learning strategy and 3D-aware convolutions to generate photorealistic, multiview consistent images with high quality, advancing 3D-aware GAN capabilities.

Contribution

The paper proposes a new 3D-to-2D imitation learning approach and incorporates 3D-aware convolutions, significantly improving image quality and 3D consistency in 3D-aware GANs.

Findings

Achieves state-of-the-art FID scores of 5.4 on FFHQ and 4.3 on AFHQ-v2 Cats.

Outperforms existing 3D-aware GANs with direct 3D rendering.

Close to the performance of methods using 2D super-resolution.

Abstract

Generating images with both photorealism and multiview 3D consistency is crucial for 3D-aware GANs, yet existing methods struggle to achieve them simultaneously. Improving the photorealism via CNN-based 2D super-resolution can break the strict 3D consistency, while keeping the 3D consistency by learning high-resolution 3D representations for direct rendering often compromises image quality. In this paper, we propose a novel learning strategy, namely 3D-to-2D imitation, which enables a 3D-aware GAN to generate high-quality images while maintaining their strict 3D consistency, by letting the images synthesized by the generator's 3D rendering branch to mimic those generated by its 2D super-resolution branch. We also introduce 3D-aware convolutions into the generator for better 3D representation learning, which further improves the image generation quality. With the above strategies, our method reaches FID scores of 5.4 and 4.3 on FFHQ and AFHQ-v2 Cats, respectively, at 512x512 resolution, largely outperforming existing 3D-aware GANs using direct 3D rendering and coming very close to the previous state-of-the-art method that leverages 2D super-resolution. Project website: https://seanchenxy.github.io/Mimic3DWeb.