Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation

TL;DR

Mosaic3D introduces a large-scale 3D dataset and a novel foundation model for open-vocabulary 3D segmentation, leveraging advanced data generation and contrastive learning to improve scene understanding.

Contribution

The paper presents Mosaic3D, a new dataset of 5.6 million mask-text pairs and a foundation model that advances open-vocabulary 3D segmentation capabilities.

Findings

Achieves state-of-the-art results on multiple 3D segmentation benchmarks.

Demonstrates the effectiveness of large-scale data and contrastive training.

Validates the approach through comprehensive ablation studies.

Abstract

We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework. Our method addresses three critical requirements for effective training: precise 3D region segmentation, comprehensive textual descriptions, and sufficient dataset scale. By leveraging state-of-the-art open-vocabulary image segmentation models and region-aware Vision-Language Models, we develop an automatic pipeline that generates high-quality 3D mask-text pairs. Applying this pipeline to multiple 3D scene datasets, we create Mosaic3D-5.6M, a dataset of over 30K annotated scenes with 5.6M mask-text pairs, significantly larger than existing datasets. Building upon this data, we propose Mosaic3D, a foundation model combining a 3D encoder trained with contrastive learning and a lightweight mask decoder for open-vocabulary 3D semantic and instance segmentation. Our approach achieves state-of-the-art results on open-vocabulary 3D semantic and instance segmentation tasks including ScanNet200, Matterport3D, and ScanNet++, with ablation studies validating the effectiveness of our large-scale training data.