ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentation

TL;DR

ELiTe introduces a novel, efficient cross-modal knowledge transfer method from vision models to LiDAR segmentation, overcoming data limitations and achieving state-of-the-art results with fewer parameters.

Contribution

The paper proposes ELiTe, a new paradigm combining multi-stage knowledge distillation, parameter-efficient fine-tuning, and pseudo-label generation for improved LiDAR semantic segmentation.

Findings

Achieves state-of-the-art results on SemanticKITTI benchmark.

Outperforms existing models with fewer parameters.

Enables real-time inference with high accuracy.

Abstract

Cross-modal knowledge transfer enhances point cloud representation learning in LiDAR semantic segmentation. Despite its potential, the \textit{weak teacher challenge} arises due to repetitive and non-diverse car camera images and sparse, inaccurate ground truth labels. To address this, we propose the Efficient Image-to-LiDAR Knowledge Transfer (ELiTe) paradigm. ELiTe introduces Patch-to-Point Multi-Stage Knowledge Distillation, transferring comprehensive knowledge from the Vision Foundation Model (VFM), extensively trained on diverse open-world images. This enables effective knowledge transfer to a lightweight student model across modalities. ELiTe employs Parameter-Efficient Fine-Tuning to strengthen the VFM teacher and expedite large-scale model training with minimal costs. Additionally, we introduce the Segment Anything Model based Pseudo-Label Generation approach to enhance low-quality image labels, facilitating robust semantic representations. Efficient knowledge transfer in ELiTe yields state-of-the-art results on the SemanticKITTI benchmark, outperforming real-time inference models. Our approach achieves this with significantly fewer parameters, confirming its effectiveness and efficiency.