GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

TL;DR

This paper introduces GRAM, a 3D-aware image generation method that uses learned radiance manifolds to improve detail and consistency, overcoming volumetric sampling limitations of previous NeRF-based models.

Contribution

The paper proposes a novel radiance manifold approach that regulates point sampling on implicit surfaces, enabling high-quality, detailed, and 3D-consistent image generation.

Findings

Produces highly realistic images with fine details

Achieves strong 3D consistency in generated images

Reduces memory and computation costs compared to volumetric methods

Abstract

3D-aware image generative modeling aims to generate 3D-consistent images with explicitly controllable camera poses. Recent works have shown promising results by training neural radiance field (NeRF) generators on unstructured 2D images, but still can not generate highly-realistic images with fine details. A critical reason is that the high memory and computation cost of volumetric representation learning greatly restricts the number of point samples for radiance integration during training. Deficient sampling not only limits the expressive power of the generator to handle fine details but also impedes effective GAN training due to the noise caused by unstable Monte Carlo sampling. We propose a novel approach that regulates point sampling and radiance field learning on 2D manifolds, embodied as a set of learned implicit surfaces in the 3D volume. For each viewing ray, we calculate ray-surface intersections and accumulate their radiance generated by the network. By training and rendering such radiance manifolds, our generator can produce high quality images with realistic fine details and strong visual 3D consistency.