3D representation in 512-Byte:Variational tokenizer is the key for autoregressive 3D generation

TL;DR

This paper introduces the Variational Tokenizer (VAT), a novel method for efficient 3D data compression and autoregressive generation, enabling high-fidelity 3D shape synthesis with significant reduction in data size.

Contribution

The paper proposes VAT, a hierarchical variational tokenizer that transforms unordered 3D data into compact tokens suitable for autoregressive modeling, addressing the challenge of 3D data's lack of inherent order.

Findings

VAT achieves up to 250x compression of 3D meshes.

VAT outperforms existing methods in quality and efficiency.

VAT maintains high F-score at extreme compression levels.

Abstract

Autoregressive transformers have revolutionized high-fidelity image generation. One crucial ingredient lies in the tokenizer, which compresses high-resolution image patches into manageable discrete tokens with a scanning or hierarchical order suitable for large language models. Extending these tokenizers to 3D generation, however, presents a significant challenge: unlike image patches that naturally exhibit spatial sequence and multi-scale relationships, 3D data lacks an inherent order, making it difficult to compress into fewer tokens while preserving structural details. To address this, we introduce the Variational Tokenizer (VAT), which transforms unordered 3D data into compact latent tokens with an implicit hierarchy, suited for efficient and high-fidelity coarse-to-fine autoregressive modeling. VAT begins with an in-context transformer, which compress numerous unordered 3D features into a reduced token set with minimal information loss. This latent space is then mapped to a Gaussian distribution for residual quantization, with token counts progressively increasing across scales. In this way, tokens at different scales naturally establish the interconnections by allocating themselves into different subspaces within the same Gaussian distribution, facilitating discrete modeling of token relationships across scales. During the decoding phase, a high-resolution triplane is utilized to convert these compact latent tokens into detailed 3D shapes. Extensive experiments demonstrate that VAT enables scalable and efficient 3D generation, outperforming existing methods in quality, efficiency, and generalization. Remarkably, VAT achieves up to a 250x compression, reducing a 1MB mesh to just 3.9KB with a 96% F-score, and can further compress to 256 int8 tokens, achieving a 2000x reduction while maintaining a 92% F-score.