TIGaussian: Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment

TL;DR

TIGaussian introduces a novel framework leveraging 3D Gaussian Splatting and multi-branch tokenization to improve cross-modal alignment between 3D data, images, and text, achieving state-of-the-art results in 3D-related tasks.

Contribution

The paper proposes a multi-branch 3DGS tokenizer and bidirectional cross-modal alignment strategies to enhance 3D feature extraction and modality bridging in visual-language models.

Findings

Achieves state-of-the-art performance on multiple 3D-related tasks.

Effectively resolves perspective ambiguity using diffusion priors.

Improves 3D and cross-modal feature alignment accuracy.

Abstract

While visual-language models have profoundly linked features between texts and images, the incorporation of 3D modality data, such as point clouds and 3D Gaussians, further enables pretraining for 3D-related tasks, e.g., cross-modal retrieval, zero-shot classification, and scene recognition. As challenges remain in extracting 3D modal features and bridging the gap between different modalities, we propose TIGaussian, a framework that harnesses 3D Gaussian Splatting (3DGS) characteristics to strengthen cross-modality alignment through multi-branch 3DGS tokenizer and modality-specific 3D feature alignment strategies. Specifically, our multi-branch 3DGS tokenizer decouples the intrinsic properties of 3DGS structures into compact latent representations, enabling more generalizable feature extraction. To further bridge the modality gap, we develop a bidirectional cross-modal alignment strategies: a multi-view feature fusion mechanism that leverages diffusion priors to resolve perspective ambiguity in image-3D alignment, while a text-3D projection module adaptively maps 3D features to text embedding space for better text-3D alignment. Extensive experiments on various datasets demonstrate the state-of-the-art performance of TIGaussian in multiple tasks.