BallGAN: 3D-aware Image Synthesis with a Spherical Background

TL;DR

BallGAN introduces a spherical background model to improve 3D-aware image synthesis, resulting in more realistic geometry, stable training, and flexible background rendering.

Contribution

It proposes a novel spherical background approximation in 3D-aware GANs, enhancing geometry realism and training stability.

Findings

More accurate 3D geometry with better consistency

Enhanced training stability compared to previous methods

Ability to render foregrounds on arbitrary backgrounds

Abstract

3D-aware GANs aim to synthesize realistic 3D scenes such that they can be rendered in arbitrary perspectives to produce images. Although previous methods produce realistic images, they suffer from unstable training or degenerate solutions where the 3D geometry is unnatural. We hypothesize that the 3D geometry is underdetermined due to the insufficient constraint, i.e., being classified as real image to the discriminator is not enough. To solve this problem, we propose to approximate the background as a spherical surface and represent a scene as a union of the foreground placed in the sphere and the thin spherical background. It reduces the degree of freedom in the background field. Accordingly, we modify the volume rendering equation and incorporate dedicated constraints to design a novel 3D-aware GAN framework named BallGAN. BallGAN has multiple advantages as follows. 1) It produces more reasonable 3D geometry; the images of a scene across different viewpoints have better photometric consistency and fidelity than the state-of-the-art methods. 2) The training becomes much more stable. 3) The foreground can be separately rendered on top of different arbitrary backgrounds.