Can3Tok: Canonical 3D Tokenization and Latent Modeling of Scene-Level 3D Gaussians

TL;DR

Can3Tok introduces a novel 3D scene-level VAE that effectively encodes complex scene data into low-dimensional representations, enabling scalable and generalizable 3D scene generation and cross-modal applications.

Contribution

It is the first 3D scene-level VAE capable of encoding numerous Gaussian primitives into a low-dimensional latent space, addressing scale inconsistency and enabling scene-level generative modeling.

Findings

Can3Tok successfully generalizes to novel 3D scenes.

Compared methods fail to converge or generalize.

Demonstrates effective image-to-3D and text-to-3D scene generation.

Abstract

3D generation has made significant progress, however, it still largely remains at the object-level. Feedforward 3D scene-level generation has been rarely explored due to the lack of models capable of scaling-up latent representation learning on 3D scene-level data. Unlike object-level generative models, which are trained on well-labeled 3D data in a bounded canonical space, scene-level generations with 3D scenes represented by 3D Gaussian Splatting (3DGS) are unbounded and exhibit scale inconsistency across different scenes, making unified latent representation learning for generative purposes extremely challenging. In this paper, we introduce Can3Tok, the first 3D scene-level variational autoencoder (VAE) capable of encoding a large number of Gaussian primitives into a low-dimensional latent embedding, which effectively captures both semantic and spatial information of the inputs. Beyond model design, we propose a general pipeline for 3D scene data processing to address scale inconsistency issue. We validate our method on the recent scene-level 3D dataset DL3DV-10K, where we found that only Can3Tok successfully generalizes to novel 3D scenes, while compared methods fail to converge on even a few hundred scene inputs during training and exhibit zero generalization ability during inference. Finally, we demonstrate image-to-3DGS and text-to-3DGS generation as our applications to demonstrate its ability to facilitate downstream generation tasks.