CLIP-GS: Unifying Vision-Language Representation with 3D Gaussian Splatting

TL;DR

CLIP-GS introduces a unified 3D multimodal learning framework using 3D Gaussian Splatting and CLIP, enabling improved 3D understanding and retrieval by integrating texture and shape information.

Contribution

The paper proposes CLIP-GS, a novel framework that combines 3D Gaussian Splatting with CLIP for unified vision-language 3D representation learning, introducing the GS Tokenizer and new training strategies.

Findings

Outperforms point cloud models on 3D tasks

Enables zero-shot and few-shot 3D classification

Achieves versatile multimodal retrieval results

Abstract

Recent works in 3D multimodal learning have made remarkable progress. However, typically 3D multimodal models are only capable of handling point clouds. Compared to the emerging 3D representation technique, 3D Gaussian Splatting (3DGS), the spatially sparse point cloud cannot depict the texture information of 3D objects, resulting in inferior reconstruction capabilities. This limitation constrains the potential of point cloud-based 3D multimodal representation learning. In this paper, we present CLIP-GS, a novel multimodal representation learning framework grounded in 3DGS. We introduce the GS Tokenizer to generate serialized gaussian tokens, which are then processed through transformer layers pre-initialized with weights from point cloud models, resulting in the 3DGS embeddings. CLIP-GS leverages contrastive loss between 3DGS and the visual-text embeddings of CLIP, and we introduce an image voting loss to guide the directionality and convergence of gradient optimization. Furthermore, we develop an efficient way to generate triplets of 3DGS, images, and text, facilitating CLIP-GS in learning unified multimodal representations. Leveraging the well-aligned multimodal representations, CLIP-GS demonstrates versatility and outperforms point cloud-based models on various 3D tasks, including multimodal retrieval, zero-shot, and few-shot classification.