SceneTok: A Compressed, Diffusable Token Space for 3D Scenes

TL;DR

SceneTok introduces a highly compressed, diffusable token-based representation for 3D scenes that achieves state-of-the-art reconstruction quality and enables rapid scene generation, surpassing existing methods in efficiency and flexibility.

Contribution

The paper presents the first scene encoding method using permutation-invariant tokens that are disentangled from spatial grids, enabling efficient, high-quality 3D scene reconstruction and generation.

Findings

Compression is 1-3 orders of magnitude better than previous methods.

Achieves state-of-the-art reconstruction quality.

Scene generation is 5 seconds with improved quality-speed trade-off.

Abstract

We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens. Existing approaches for 3D scene representation and generation commonly use 3D data structures or view-aligned fields. In contrast, we introduce the first method that encodes scene information into a small set of permutation-invariant tokens that is disentangled from the spatial grid. The scene tokens are predicted by a multi-view tokenizer given many context views and rendered into novel views by employing a light-weight rectified flow decoder. We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality. Further, our representation can be rendered from novel trajectories, including ones deviating from the input trajectory, and we show that the decoder gracefully handles uncertainty. Finally, the highly-compressed set of unstructured latent scene tokens enables simple and efficient scene generation in 5 seconds, achieving a much better quality-speed trade-off than previous paradigms.