UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mapping

TL;DR

This paper introduces UVGS, a novel 2D UV mapping-based representation for 3D Gaussian Splatting that enables leveraging 2D diffusion models for efficient 3D scene generation and editing.

Contribution

The paper proposes UVGS, a structured 2D representation of 3D Gaussian Splatting that allows direct application of 2D models like diffusion for 3D generation.

Findings

UVGS can be compressed into a 3-channel shared feature space.

Pretrained 2D diffusion models can be directly applied to UVGS without retraining.

UVGS enables scalable, high-quality 3D scene generation and editing.

Abstract

3D Gaussian Splatting (3DGS) has demonstrated superior quality in modeling 3D objects and scenes. However, generating 3DGS remains challenging due to their discrete, unstructured, and permutation-invariant nature. In this work, we present a simple yet effective method to overcome these challenges. We utilize spherical mapping to transform 3DGS into a structured 2D representation, termed UVGS. UVGS can be viewed as multi-channel images, with feature dimensions as a concatenation of Gaussian attributes such as position, scale, color, opacity, and rotation. We further find that these heterogeneous features can be compressed into a lower-dimensional (e.g., 3-channel) shared feature space using a carefully designed multi-branch network. The compressed UVGS can be treated as typical RGB images. Remarkably, we discover that typical VAEs trained with latent diffusion models can directly generalize to this new representation without additional training. Our novel representation makes it effortless to leverage foundational 2D models, such as diffusion models, to directly model 3DGS. Additionally, one can simply increase the 2D UV resolution to accommodate more Gaussians, making UVGS a scalable solution compared to typical 3D backbones. This approach immediately unlocks various novel generation applications of 3DGS by inherently utilizing the already developed superior 2D generation capabilities. In our experiments, we demonstrate various unconditional, conditional generation, and inpainting applications of 3DGS based on diffusion models, which were previously non-trivial.