Segment Any Point Cloud Sequences by Distilling Vision Foundation Models

TL;DR

This paper introduces Seal, a framework that leverages vision foundation models to efficiently and consistently segment diverse automotive point cloud sequences without requiring annotations, demonstrating superior performance across multiple datasets.

Contribution

Seal is the first method to distill vision foundation models into point cloud segmentation, enabling scalable, consistent, and generalizable knowledge transfer for diverse datasets.

Findings

Achieves 45.0% mIoU on nuScenes, surpassing prior methods.

Outperforms existing methods in 20 few-shot fine-tuning tasks.

Demonstrates effectiveness across 11 diverse point cloud datasets.

Abstract

Recent advancements in vision foundation models (VFMs) have opened up new possibilities for versatile and efficient visual perception. In this work, we introduce Seal, a novel framework that harnesses VFMs for segmenting diverse automotive point cloud sequences. Seal exhibits three appealing properties: i) Scalability: VFMs are directly distilled into point clouds, obviating the need for annotations in either 2D or 3D during pretraining. ii) Consistency: Spatial and temporal relationships are enforced at both the camera-to-LiDAR and point-to-segment regularization stages, facilitating cross-modal representation learning. iii) Generalizability: Seal enables knowledge transfer in an off-the-shelf manner to downstream tasks involving diverse point clouds, including those from real/synthetic, low/high-resolution, large/small-scale, and clean/corrupted datasets. Extensive experiments conducted on eleven different point cloud datasets showcase the effectiveness and superiority of Seal. Notably, Seal achieves a remarkable 45.0% mIoU on nuScenes after linear probing, surpassing random initialization by 36.9% mIoU and outperforming prior arts by 6.1% mIoU. Moreover, Seal demonstrates significant performance gains over existing methods across 20 different few-shot fine-tuning tasks on all eleven tested point cloud datasets.