GSGAN: Adversarial Learning for Hierarchical Generation of 3D Gaussian Splats

TL;DR

GSGAN introduces a hierarchical Gaussian representation for 3D GANs, enabling faster rendering and improved stability, with the ability to generate detailed 3D scenes efficiently.

Contribution

The paper proposes a novel hierarchical multi-scale Gaussian generator architecture for 3D GANs, addressing training instability and scale adjustment issues.

Findings

Achieves 100x faster rendering speed than state-of-the-art 3D GANs.

Maintains comparable 3D generation quality.

Demonstrates effective modeling of coarse and fine 3D scene details.

Abstract

Most advances in 3D Generative Adversarial Networks (3D GANs) largely depend on ray casting-based volume rendering, which incurs demanding rendering costs. One promising alternative is rasterization-based 3D Gaussian Splatting (3D-GS), providing a much faster rendering speed and explicit 3D representation. In this paper, we exploit Gaussian as a 3D representation for 3D GANs by leveraging its efficient and explicit characteristics. However, in an adversarial framework, we observe that a na\"ive generator architecture suffers from training instability and lacks the capability to adjust the scale of Gaussians. This leads to model divergence and visual artifacts due to the absence of proper guidance for initialized positions of Gaussians and densification to manage their scales adaptively. To address these issues, we introduce a generator architecture with a hierarchical multi-scale Gaussian representation that effectively regularizes the position and scale of generated Gaussians. Specifically, we design a hierarchy of Gaussians where finer-level Gaussians are parameterized by their coarser-level counterparts; the position of finer-level Gaussians would be located near their coarser-level counterparts, and the scale would monotonically decrease as the level becomes finer, modeling both coarse and fine details of the 3D scene. Experimental results demonstrate that ours achieves a significantly faster rendering speed (x100) compared to state-of-the-art 3D consistent GANs with comparable 3D generation capability. Project page: https://hse1032.github.io/gsgan.