3D Shape Tokenization via Latent Flow Matching

TL;DR

This paper presents a novel 3D shape tokenization method using latent flow matching, modeling 3D surfaces as probability densities, enabling efficient learning and zero-shot geometry estimation with minimal data preprocessing.

Contribution

Introduces a flow-matching based latent 3D representation that is continuous, compact, and suitable for machine learning, with capabilities for zero-shot surface normal and deformation estimation.

Findings

Achieves competitive performance on multiple 3D tasks

Requires less preprocessing than existing methods

Enables zero-shot estimation of surface properties

Abstract

We introduce a latent 3D representation that models 3D surfaces as probability density functions in 3D, i.e., p(x,y,z), with flow-matching. Our representation is specifically designed for consumption by machine learning models, offering continuity and compactness by construction while requiring only point clouds and minimal data preprocessing. Despite being a data-driven method, our use of flow matching in the 3D space enables interesting geometry properties, including the capabilities to perform zero-shot estimation of surface normal and deformation field. We evaluate with several machine learning tasks, including 3D-CLIP, unconditional generative models, single-image conditioned generative model, and intersection-point estimation. Across all experiments, our models achieve competitive performance to existing baselines, while requiring less preprocessing and auxiliary information from training data.