Contrastive Language-Colored Pointmap Pretraining for Unified 3D Scene Understanding

TL;DR

UniScene3D is a transformer-based encoder that learns unified 3D scene representations from multi-view colored pointmaps, achieving state-of-the-art results in various scene understanding tasks.

Contribution

The paper introduces novel cross-view geometric and grounded view alignment techniques for robust 3D scene pretraining with a unified transformer-based model.

Findings

State-of-the-art performance in viewpoint grounding and scene retrieval.

Effective low-shot and task-specific fine-tuning results.

Unified 3D scene understanding from multi-view pointmaps.

Abstract

Pretraining 3D encoders by aligning with Contrastive Language Image Pretraining (CLIP) has emerged as a promising direction to learn generalizable representations for 3D scene understanding. In this paper, we propose UniScene3D, a transformer-based encoder that learns unified scene representations from multi-view colored pointmaps, jointly modeling image appearance and geometry. For robust colored pointmap representation learning, we introduce novel cross-view geometric alignment and grounded view alignment to enforce cross-view geometry and semantic consistency. Extensive low-shot and task-specific fine-tuning evaluations on viewpoint grounding, scene retrieval, scene type classification, and 3D VQA demonstrate our state-of-the-art performance. These results highlight the effectiveness of our approach for unified 3D scene understanding. https://yebulabula.github.io/UniScene3D/