VQ3D: Learning a 3D-Aware Generative Model on ImageNet

TL;DR

VQ3D introduces a novel 3D-aware generative model that leverages a NeRF-based decoder within a two-stage vector-quantized autoencoder, enabling high-quality 3D-aware image generation and reconstruction on the complex ImageNet dataset.

Contribution

The paper presents VQ3D, a new model that effectively generates and reconstructs 3D-aware images from large, diverse datasets like ImageNet, overcoming previous limitations on complex data.

Findings

Achieves a 16.8 FID score on ImageNet, significantly better than baseline.

Capable of reconstructing images with adjustable camera positions.

Generates diverse 3D-aware images across 1000 classes.

Abstract

Recent work has shown the possibility of training generative models of 3D content from 2D image collections on small datasets corresponding to a single object class, such as human faces, animal faces, or cars. However, these models struggle on larger, more complex datasets. To model diverse and unconstrained image collections such as ImageNet, we present VQ3D, which introduces a NeRF-based decoder into a two-stage vector-quantized autoencoder. Our Stage 1 allows for the reconstruction of an input image and the ability to change the camera position around the image, and our Stage 2 allows for the generation of new 3D scenes. VQ3D is capable of generating and reconstructing 3D-aware images from the 1000-class ImageNet dataset of 1.2 million training images. We achieve an ImageNet generation FID score of 16.8, compared to 69.8 for the next best baseline method.