Multi-modal NeRF Self-Supervision for LiDAR Semantic Segmentation

TL;DR

This paper introduces a semi-supervised method that combines NeRF-based self-supervision and foundation model-derived pseudo-labels to improve LiDAR semantic segmentation without requiring extensive annotations.

Contribution

It proposes a novel semi-supervised framework leveraging NeRF and foundation models to enhance LiDAR segmentation, addressing domain adaptation and annotation limitations.

Findings

Improves segmentation accuracy on nuScenes, SemanticKITTI, ScribbleKITTI

Effectively utilizes unlabeled LiDAR data with self-supervision

Reduces dependence on extensive manual annotations

Abstract

LiDAR Semantic Segmentation is a fundamental task in autonomous driving perception consisting of associating each LiDAR point to a semantic label. Fully-supervised models have widely tackled this task, but they require labels for each scan, which either limits their domain or requires impractical amounts of expensive annotations. Camera images, which are generally recorded alongside LiDAR pointclouds, can be processed by the widely available 2D foundation models, which are generic and dataset-agnostic. However, distilling knowledge from 2D data to improve LiDAR perception raises domain adaptation challenges. For example, the classical perspective projection suffers from the parallax effect produced by the position shift between both sensors at their respective capture times. We propose a Semi-Supervised Learning setup to leverage unlabeled LiDAR pointclouds alongside distilled knowledge from the camera images. To self-supervise our model on the unlabeled scans, we add an auxiliary NeRF head and cast rays from the camera viewpoint over the unlabeled voxel features. The NeRF head predicts densities and semantic logits at each sampled ray location which are used for rendering pixel semantics. Concurrently, we query the Segment-Anything (SAM) foundation model with the camera image to generate a set of unlabeled generic masks. We fuse the masks with the rendered pixel semantics from LiDAR to produce pseudo-labels that supervise the pixel predictions. During inference, we drop the NeRF head and run our model with only LiDAR. We show the effectiveness of our approach in three public LiDAR Semantic Segmentation benchmarks: nuScenes, SemanticKITTI and ScribbleKITTI.