SAM-Guided Masked Token Prediction for 3D Scene Understanding

TL;DR

This paper introduces a SAM-guided tokenization and masked feature prediction framework to improve 3D scene understanding by better aligning 2D-3D representations and addressing dataset imbalances, achieving state-of-the-art results.

Contribution

The paper proposes a novel SAM-guided tokenization method and a two-stage masked token prediction framework for enhanced 3D scene understanding, addressing alignment and long-tail challenges.

Findings

Significant performance improvements on SUN RGB-D, ScanNet, and S3DIS datasets.

Establishment of new benchmarks in 3D object detection and semantic segmentation.

Effective handling of long-tail distribution in 3D datasets.

Abstract

Foundation models have significantly enhanced 2D task performance, and recent works like Bridge3D have successfully applied these models to improve 3D scene understanding through knowledge distillation, marking considerable advancements. Nonetheless, challenges such as the misalignment between 2D and 3D representations and the persistent long-tail distribution in 3D datasets still restrict the effectiveness of knowledge distillation from 2D to 3D using foundation models. To tackle these issues, we introduce a novel SAM-guided tokenization method that seamlessly aligns 3D transformer structures with region-level knowledge distillation, replacing the traditional KNN-based tokenization techniques. Additionally, we implement a group-balanced re-weighting strategy to effectively address the long-tail problem in knowledge distillation. Furthermore, inspired by the recent success of masked feature prediction, our framework incorporates a two-stage masked token prediction process in which the student model predicts both the global embeddings and the token-wise local embeddings derived from the teacher models trained in the first stage. Our methodology has been validated across multiple datasets, including SUN RGB-D, ScanNet, and S3DIS, for tasks like 3D object detection and semantic segmentation. The results demonstrate significant improvements over current State-of-the-art self-supervised methods, establishing new benchmarks in this field.