Feasibility of Indoor Frame-Wise Lidar Semantic Segmentation via Distillation from Visual Foundation Model

TL;DR

This paper explores using visual foundation models to distill semantic segmentation knowledge from images to indoor lidar scans, enabling effective indoor scene understanding without extensive manual labeling.

Contribution

It demonstrates the feasibility of cross-modal distillation from images to lidar for indoor scenes, achieving promising segmentation accuracy without manual annotations.

Findings

Distilled model achieves up to 56% mIoU with pseudo-labels.

Achieves around 36% mIoU with real labels.

Validates approach on indoor SLAM datasets with a new lidar dataset.

Abstract

Frame-wise semantic segmentation of indoor lidar scans is a fundamental step toward higher-level 3D scene understanding and mapping applications. However, acquiring frame-wise ground truth for training deep learning models is costly and time-consuming. This challenge is largely addressed, for imagery, by Visual Foundation Models (VFMs) which segment image frames. The same VFMs may be used to train a lidar scan frame segmentation model via a 2D-to-3D distillation pipeline. The success of such distillation has been shown for autonomous driving scenes, but not yet for indoor scenes. Here, we study the feasibility of repeating this success for indoor scenes, in a frame-wise distillation manner by coupling each lidar scan with a VFM-processed camera image. The evaluation is done using indoor SLAM datasets, where pseudo-labels are used for downstream evaluation. Also, a small manually annotated lidar dataset is provided for validation, as there are no other lidar frame-wise indoor datasets with semantics. Results show that the distilled model achieves up to 56% mIoU under pseudo-label evaluation and around 36% mIoU with real-label, demonstrating the feasibility of cross-modal distillation for indoor lidar semantic segmentation without manual annotations.