Native and Compact Structured Latents for 3D Generation

TL;DR

This paper introduces O-Voxel, a novel sparse voxel structure for 3D generative modeling that captures complex topologies and detailed surface attributes, enabling high-quality, efficient 3D asset generation.

Contribution

The paper presents O-Voxel, a new omni-voxel representation and a Sparse Compression VAE for improved 3D asset modeling with complex topologies and rich surface details.

Findings

O-Voxel effectively models arbitrary topologies.

Large-scale flow-matching models generate high-quality 3D assets.

Inference remains efficient despite model scale.

Abstract

Recent advancements in 3D generative modeling have significantly improved the generation realism, yet the field is still hampered by existing representations, which struggle to capture assets with complex topologies and detailed appearance. This paper present an approach for learning a structured latent representation from native 3D data to address this challenge. At its core is a new sparse voxel structure called O-Voxel, an omni-voxel representation that encodes both geometry and appearance. O-Voxel can robustly model arbitrary topology, including open, non-manifold, and fully-enclosed surfaces, while capturing comprehensive surface attributes beyond texture color, such as physically-based rendering parameters. Based on O-Voxel, we design a Sparse Compression VAE which provides a high spatial compression rate and a compact latent space. We train large-scale flow-matching models comprising 4B parameters for 3D generation using diverse public 3D asset datasets. Despite their scale, inference remains highly efficient. Meanwhile, the geometry and material quality of our generated assets far exceed those of existing models. We believe our approach offers a significant advancement in 3D generative modeling.