SYM3D: Learning Symmetric Triplanes for Better 3D-Awareness of GANs

TL;DR

SYM3D introduces a symmetry-aware 3D GAN that leverages reflectional symmetry and view-aware attention to improve 3D asset generation from limited data, outperforming existing methods in detail and consistency.

Contribution

The paper presents SYM3D, a novel 3D-aware GAN that incorporates symmetry regularization and view-aware attention, enabling high-quality 3D asset synthesis from single-view images.

Findings

Superior geometry and texture capture on synthetic datasets.

Effective reduction of artifacts in text-to-3D modeling.

Robust performance on real-world ABO-Chair dataset.

Abstract

Despite the growing success of 3D-aware GANs, which can be trained on 2D images to generate high-quality 3D assets, they still rely on multi-view images with camera annotations to synthesize sufficient details from all viewing directions. However, the scarce availability of calibrated multi-view image datasets, especially in comparison to single-view images, has limited the potential of 3D GANs. Moreover, while bypassing camera pose annotations with a camera distribution constraint reduces dependence on exact camera parameters, it still struggles to generate a consistent orientation of 3D assets. To this end, we propose SYM3D, a novel 3D-aware GAN designed to leverage the prevalent reflectional symmetry structure found in natural and man-made objects, alongside a proposed view-aware spatial attention mechanism in learning the 3D representation. We evaluate SYM3D on both synthetic (ShapeNet Chairs, Cars, and Airplanes) and real-world datasets (ABO-Chair), demonstrating its superior performance in capturing detailed geometry and texture, even when trained on only single-view images. Finally, we demonstrate the effectiveness of incorporating symmetry regularization in helping reduce artifacts in the modeling of 3D assets in the text-to-3D task. Project is at \url{https://jingyang2017.github.io/sym3d.github.io/}