Domain Adaptation-Based Crossmodal Knowledge Distillation for 3D Semantic Segmentation

TL;DR

This paper introduces two crossmodal knowledge distillation methods that leverage synchronized 2D camera images to improve 3D LiDAR semantic segmentation without requiring extensive 3D annotations, using domain adaptation and feature alignment techniques.

Contribution

It proposes novel unsupervised domain adaptation and feature distillation methods that utilize 2D-3D data correspondence to reduce annotation costs in 3D semantic segmentation.

Findings

Outperforms state-of-the-art methods in 3D segmentation accuracy.

Effectively leverages synchronized 2D images for 3D model training.

Demonstrates robustness across different autonomous driving datasets.

Abstract

Semantic segmentation of 3D LiDAR data plays a pivotal role in autonomous driving. Traditional approaches rely on extensive annotated data for point cloud analysis, incurring high costs and time investments. In contrast, realworld image datasets offer abundant availability and substantial scale. To mitigate the burden of annotating 3D LiDAR point clouds, we propose two crossmodal knowledge distillation methods: Unsupervised Domain Adaptation Knowledge Distillation (UDAKD) and Feature and Semantic-based Knowledge Distillation (FSKD). Leveraging readily available spatio-temporally synchronized data from cameras and LiDARs in autonomous driving scenarios, we directly apply a pretrained 2D image model to unlabeled 2D data. Through crossmodal knowledge distillation with known 2D-3D correspondence, we actively align the output of the 3D network with the corresponding points of the 2D network, thereby obviating the necessity for 3D annotations. Our focus is on preserving modality-general information while filtering out modality-specific details during crossmodal distillation. To achieve this, we deploy self-calibrated convolution on 3D point clouds as the foundation of our domain adaptation module. Rigorous experimentation validates the effectiveness of our proposed methods, consistently surpassing the performance of state-of-the-art approaches in the field.

Figures

Tables

Table 1. TABLE I: Few-Shot 3D Semantic Segmentation Results

Method	nuScenes		KITTI
	1%	100%	1%	100%
Random	31.8	74.2	39.5	53.1
I2P-MAE [32]	34.2	74.2	40.9	53.0
PPKT [8]	37.8	74.4	44.2	52.9
SLidR [6]	38.3	74.7	44.6	53.2
UDAKD (Ours)	39.3	75.0	45.1	54.0
FSKD (Ours)	44.7	74.9	46.2	53.8

Table 2. TABLE II: Zero-Shot 3D Semantic Segmentation Results

Method	mIoU
OpenScene [36]	41.5
MVF [54]	55.8
2DPASS [53]	56.8
FSKD (Ours)	63.7

Table 3. TABLE III: Segmentation Results with Varying Annotation Availability

Method	0%	1%	5%	10%	25%	100%
SLidR [6]	4.1	32.9	58.2	66.6	71.2	78.1
FSKD (Ours)	63.7	62.0	66.3	69.6	71.2	78.6

Table 4. TABLE IV: Zero-Shot Domain Adaptation Results

Method	noise			pedestrian
	debris	pshb. pllb. obj.	bicycle rack	adult	const. wkr.
FSKD*	0.5	2.0	6.7	47.9	10.6
SegNet*	2.5	9.1	6.3	76.0	11.7

Table 5. TABLE V: Ablation Study of UDAKD

Method	Sp.	DA	mIoU
UDAKD	✔	✔	39.3
UDAKD w/o sp.	✗	✔	38.6 (-0.7)
SLidR [6]	✔	✗	38.3 (-1.0)
SLidR [6] w/o sp.	✗	✗	36.6 (-2.7)

Table 6. TABLE VI: Ablation Study of FSKD

Method	mIoU
FSKD	63.7
FSKD w/o soft labels	55.1 (-8.6)
FSKD w/o feature-based KD	55.3 (-8.4)
FSKD w/o semantic-based KD	48.0 (-15.7)
FSKD w/o DA module	48.5 (-15.2)

Equations

$$\mathcal{F}=e_{{\theta }_e}\left(\mathcal{I}\right).$$

$$\mathcal{G}=h_{{\omega }_h}\left(\mathcal{P}\right).$$

$$\mathcal{S}=d_{{\theta }_d}\left(\mathcal{F}\right),\mathcal{T}=k_{{\omega }_k}\left(\mathcal{G}\right).$$

$${\mathcal{G}}^{\prime }=m_{\lambda }\left(\mathcal{G}\right).$$

$${\mathbf{f}}_i=\frac{1}{\left|{\mathcal{A}}_i\right|}\sum _{\text{pix}\in {\mathcal{A}}_i}\mathcal{F}(\text{pix}),{\mathbf{g}}_i=\frac{1}{\left|{\mathcal{B}}_i\right|}\sum _{\text{pnt}\in {\mathcal{B}}_i}{\mathcal{G}}^{\prime }(\text{pnt}),$$

$$\begin{array}{rl}& {\mathcal{L}}_{\text{feat}}^{\text{UDAKD}}({\theta }^{\text{hd}},{\omega }_h,\lambda )\\ =& -\sum _i\log \left[\frac{\exp \left(\frac{{{\mathbf{g}}_i}^{\top }{\mathbf{f}}_i}{\tau }\right)}{{\sum }_{i^{\prime }\ne i}\exp \left(\frac{{{\mathbf{g}}_i}^{\top }{\mathbf{f}}_{i^{\prime }}}{\tau }\right)+\exp \left(\frac{{{\mathbf{g}}_i}^{\top }{\mathbf{f}}_i}{\tau }\right)}\right],\end{array}$$

$${\mathcal{L}}_{\text{feat}}^{\text{FSKD}}({\omega }_h,\lambda )=MSE\left({\mathcal{G}}^{\prime },\mathcal{F}\right).$$

$${\mathcal{L}}_{\text{sem}}^{\text{FSKD}}(\omega ,\lambda )=KL\left(\mathcal{T}\Vert \mathcal{S}\right).$$

Full text

I Introduction

Recent years have witnessed a surge in perception algorithms for processing LiDAR point cloud data, supported by diverse open benchmarks [1, 2, 3, 4]. Despite these advances, the efficacy of such techniques is often impeded by their reliance on meticulously annotated data, leading to time-consuming and labor-intensive annotation procedures [5, 6]. Moreover, existing domain adaptation methods have shown limited effectiveness in handling the complexities of 3D point clouds, necessitating separate training of 3D networks for different LiDAR configurations, thereby exacerbating the annotation burden [7]. Consequently, a pressing demand has arisen to address the challenge of annotating LiDAR point clouds.

To reduce the annotation burden while enhancing semantic segmentation learning for 3D networks, we employ crossmodal knowledge distillation from 2D to 3D, leveraging paired image and LiDAR data. We introduce two methods: Unsupervised Domain Adaptation Knowledge Distillation (UDAKD) and Feature and Semantic-based Knowledge Distillation (FSKD), as depicted in Fig. 1.

In UDAKD, akin to PPKT [8] and SLidR [6], we bypass the need for the original 2D training dataset by directly aligning 2D and 3D network features on multimodal data, providing a pretraining approach for the 3D network.

In FSKD, we leverage the 2D semantic segmentation model trained on labeled images to extract features and generate per-pixel soft labels for unlabeled target data. By aligning the features and labels from the 3D network with the corresponding 2D network points, we train a comprehensive 3D semantic segmentation model without explicit 3D annotations.

Inspired by [9], we aim to minimize modality-specific feature interference while maximizing modality-general feature extraction during crossmodal knowledge distillation. To achieve this, we design a domain adaptation module based on self-calibrated convolution [10], which has shown promise in aligning feature distributions of images with those from LiDAR point clouds [11]. We implement 3D self-calibrated convolution within the LiDAR point cloud modality.

Our main contributions can be summarized as follows:

• •

We propose a novel domain adaptation module that employs 3D self-calibrated convolution to transfer modality-general features and eliminate modality-specific details.

• •

We introduce distinct crossmodal knowledge distillation strategies for unlabeled and labeled images, enhancing 3D semantic segmentation via image-to-LiDAR knowledge distillation.

• •

Our UDAKD and FSKD methods, designed for unlabeled and labeled image scenarios respectively, achieve superior performance compared to previous state-of-the-art (SOTA) methods.

II Related Work

II-A LiDAR-Based Semantic Segmentation

Semantic segmentation networks for LiDAR data predominantly rely on the U-Net architecture [12]. Given the sparsity of point clouds, three main representations are used: i) Point-based methods process original points in continuous 3D space [13, 14, 15, 16], often suitable for small-scale synthetic or indoor point clouds. ii) Projection-based methods project point clouds onto range views [17, 18, 19, 20, 21] or bird’s-eye views [22, 23], potentially disrupting the 3D spatial structure. iii) Voxel-based methods, e.g., Minkowski U-Net [24, 25], Cylinder3D [26], AF2-S3Net [27], SPVNAS [28], and SVQNet [29], quantize 3D points into sparse voxels, convolving only on non-empty voxels to reduce computational costs. Our work adopts the voxel-based approach for efficient LiDAR point cloud feature extraction.

II-B Crossmodal Supervision between Image and Point Cloud

We utilize knowledge distillation for crossmodal supervision, transferring knowledge from a 2D image teacher network to a 3D point cloud student network, enhancing the 3D network’s performance with limited annotations.

Indoor scenario

Gupta et al. [30] transfer knowledge from annotated color images to unannotated depth or optical flow images using intermediate representations. Meyer et al. [31] introduce multimodal contrastive learning for RGB-D images when only one modality is available during inference. PPKT [8] pretrains a 3D network on unlabeled RGB-D datasets using an existing 2D network without accessing the original 2D training data. I2P-MAE [32] uses self-supervised 2D knowledge to guide the 3D masked autoencoding process.

Autonomous driving scenario

Matching 2D and 3D points in autonomous driving is challenging due to sparse LiDAR sampling and continuous motion [33]. To address this, 2D3DNet [5] uses high-confidence 2D outputs as pseudo-labels and integrates weighted voting of these outputs as semantic features for the 3D network. xMUDA and xMoSSDA [34, 35] enforce consistency between point cloud and image predictions through mutual mimicking. OpenScene [36] uses a 2D model pretrained for open-vocabulary semantic segmentation to supervise the 3D network. CMKD [11] transforms image features to resemble point clouds using a self-calibrated convolution-based domain adaptation module. SLidR [6] employs contrastive learning on SLIC superpixels [37] to address coverage issues in LiDAR point clouds. While previous works primarily enhance 2D data processing for improved teacher models, optimizing the 3D student has been somewhat overlooked. Our work fills this gap by integrating a domain adaptation module into the 3D student network.

Theoretical studies

Image2Point [38] directly transfers convolutional kernels from 2D to 3D, achieving competitive point cloud classification performance. They establish that the extent of neural collapse [39] in a model pretrained on source domain data sets the upper limit of its performance on target tasks. Inspired by [9], which proposes the modality focusing hypothesis (MFH), we emphasize modality-general features in crossmodal knowledge distillation. Our domain adaptation module transfers modality-general features while separating modality-specific features, diverging from previous methods with fewer such considerations.

III Method

III-A Problem Setup

Let $\mathbf{S}=\{\mathbf{I}_{s}\}$ be an unlabeled dataset of RGB images from a singlemodal source domain $s$, used for self-supervised pretraining of a feature extractor $e_{\theta_{e}}$ with parameters $\theta_{e}$. When labels are available, $\mathbf{S}=\{\mathbf{I}_{s},\mathbf{L}_{s}\}$ includes corresponding semantic annotations $\mathbf{L}_{s}$. In this scenario, we can pretrain a 2D semantic segmentation network $f_{\theta}=d_{\theta_{d}}\circ e_{\theta_{e}}$, where $\circ$ denotes function composition and $d_{\theta_{d}}$ is a per-pixel classifier producing soft labels for $C_{\text{cls}}$ semantic classes.

In the target domain $t$, $\mathbf{T}=\{\mathbf{I}_{t},\mathbf{P}_{t}\}$ contains synchronized images $\mathbf{I}_{t}$ and point clouds $\mathbf{P}_{t}$. Our goal is to use $\mathbf{T}$ and the 2D extractor $e_{\theta_{e}}$ to obtain a 3D feature extractor $h_{\omega_{h}}$, or with the aid of $\mathbf{T}$ and the complete 2D network $f_{\theta}$, to derive a 3D segmentation network $g_{\omega}=k_{\omega_{k}}\circ h_{\omega_{h}}$. If annotations $\mathbf{L}_{t}$ are available, we refine the 3D network to $\widetilde{g}_{\widetilde{\omega}}$ with modified structure $\widetilde{g}$ and parameters $\widetilde{\omega}$.

III-B Framework Overview

We propose two distinct approaches: Unsupervised Domain Adaptation Knowledge Distillation (UDAKD) for scenarios with unlabeled images, and Feature and Semantic-based Knowledge Distillation (FSKD) for labeled images. These methods are illustrated in Fig. 1.

Both UDAKD and FSKD use dedicated networks to extract features from 3D point clouds and 2D images. The 3D features are adapted within the domain adaptation module $m_{\lambda}$ to resemble 2D features, facilitating feature-based knowledge distillation. In FSKD, 2D and 3D features are further fed into a shared MLP classifier to generate point-wise semantic soft labels, enabling semantic-based knowledge distillation.

Similar to TPVFormer [40], our approach imposes no constraints on the number of cameras or LiDAR sensors. Distillation is performed in regions visible to both modalities.

III-C Segmentation Networks

2D feature extractor

The 2D feature extraction network $e_{\theta_{e}}$ converts an RGB image $\mathcal{I}\in\mathbb{R}^{3\times H\times W}$ into $C_{\text{img}}$-dimensional image features $\mathcal{F}\in\mathbb{R}^{C_{\text{img}}\times H\times W}$:

[TABLE]

In UDAKD, $e_{\theta_{e}}$ comprises a ResNet-50 backbone with parameters $\theta^{\text{\tiny bck}}$ and a projection head with $1\times 1$ convolutions and bilinear upsampling, parameterized by $\theta^{\text{\tiny hd}}$, to align with the input image size. For FSKD, we employ SegNet [41] as $e_{\theta_{e}}$ to reduce parameter count and computational cost while maintaining performance. Our algorithm allows flexible selection from various mature networks.

3D feature extractor

For 3D feature extraction, we use MinkUNet32 [6, 25] as $h_{\omega_{h}}$, which maps a point cloud $\mathcal{P}\in\mathbb{R}^{D\times N}$ with $N$ points and $D$ dimensions to features $\mathcal{G}\in\mathbb{R}^{C_{\text{pc}}\times N}$ with $C_{\text{pc}}$ dimensions:

[TABLE]

Shared classifier

In FSKD, we use a 2D classifier $d_{\theta_{d}}$ and a 3D classifier $k_{\omega_{k}}$ alongside their respective feature extractors to form complete 2D network $f_{\theta}=d_{\theta_{d}}\circ e_{\theta_{e}}$ and 3D network $g_{\omega}=k_{\omega_{k}}\circ h_{\omega_{h}}$. These classifiers produce soft semantic labels $\mathcal{S}\in\mathbb{R}^{C_{\text{cls}}\times H\times W}$ and $\mathcal{T}\in\mathbb{R}^{C_{\text{cls}}\times N}$ from features $\mathcal{F}$ and $\mathcal{G}$, respectively:

[TABLE]

For zero-shot domain adaptation, allowing the 3D network to learn new classes from images alone, we adopt a unified structure with shared parameters between $d_{\theta_{d}}$ and $k_{\omega_{k}}$, i.e., $k_{\omega_{k}}=d_{\theta_{d}}$. By training a modified 2D segmentation network $f_{\theta^{\prime}}^{\prime}=d_{\theta_{d}^{\prime}}^{\prime}\circ e_{\theta_{e}}$ on new images, keeping $e_{\theta_{e}}$ fixed and introducing a new classifier $d_{\theta_{d}^{\prime}}^{\prime}$, we derive a new 3D semantic segmentation network $g_{\omega^{\prime}}^{\prime}=d_{\theta_{d}^{\prime}}^{\prime}\circ m_{\lambda}\circ h_{\omega_{h}}$, capable of recognizing new classes without annotated 3D data.

Under the neural collapse phenomenon [39], where over-parameterized classifiers yield features clustering around class means, we assume complete neural collapse for both 2D and 3D extractors. Ideally, this allows a linear transformation to map 3D point cloud features to 2D features, with the classifier converting these into semantic labels. Using a shared MLP-based classifier for both networks thus minimizes feature extractor constraints and enables zero-shot learning for the 3D network. Acknowledging the impracticality of perfect neural collapse, we introduce a domain adaptation (DA) module to handle this variability, as detailed below.

III-D Domain Adaptation Module

Previous work on multimodal fusion typically involves both 3D point clouds and 2D images during training and inference, leading to high computational costs and slower inference due to separate network structures for each modality. Common solutions include early MLP fusion [34, 42, 43] or using image network outputs as additional features for 3D points [5, 44, 45, 46], followed by end-to-end training.

In our task, the absence of the image modality during inference requires the 3D network to independently perform semantic segmentation on point clouds without relying on image inputs. While during knowledge distillation, the 3D network learns solely from its 2D teacher. Hence, a dedicated domain adaptation module is essential.

We employ the domain adaptation module $m_{\lambda}$ based on self-calibrated convolution, initially proposed by [10]. Although methods like CMKD [11] and MSNetSC [47] have applied this in 2D contexts, its applicability in 3D remains unexplored. We utilize multi-layered 3D self-calibrated convolutions in $m_{\lambda}$ for both UDAKD and FSKD, converting 3D features $\mathcal{G}\in\mathbb{R}^{C_{\text{pc}}\times N}$ into pseudo-2D features $\mathcal{G}^{\prime}\in\mathbb{R}^{C_{\text{img}}\times N}$:

[TABLE]

Previous studies [5, 6] on 2D-to-3D supervision often neglect the 3D network’s effort in extracting meaningful insights from limited knowledge. The DA module is crucial and may outweigh the importance of specific structural details, as it takes the job of efficiently transforming crossmodal information, allowing the 3D network to focus on learning semantically rich features. While the DA module can vary, including MLP and vanilla convolutions, self-calibrated convolutions offer superior efficiency.

III-E Feature-Based Knowledge Distillation

Both UDAKD and FSKD entail aligning the distribution of 3D features $\mathcal{G}^{C_{\text{pc}}\times N}$ with that of 2D features $\mathcal{F}^{C_{\text{img}}\times H\times W}$ through feature-based knowledge distillation. This process enables the 3D feature extractor $h_{\omega_{h}}$ to learn to transform inputs into semantically rich representations.

In the UDAKD framework, the 3D network is trained exclusively with feature-based distillation, formalized by the loss $\mathcal{L}_{\text{\tiny kd}}^{\text{\tiny UDAKD}}=\mathcal{L}_{\text{\tiny feat}}^{\text{\tiny UDAKD}}$. The main goal is to enhance knowledge transfer from the 2D teacher to the 3D student at the feature level. Our approach resembles SLidR [6], which aligns features at the superpixel level using contrastive learning, but extends it by incorporating the domain adaptation module $m_{\lambda}$, leading to significant improvements. Before alignment, we average pool the 2D feature map $\mathcal{F}$ over pixels within SLIC [37] superpixels and the pseudo-2D feature map $\mathcal{G}^{\prime}$ over points within corresponding superpoints, yielding pooled feature vectors $\mathbf{f}_{i}$ and $\mathbf{g}_{i}$:

[TABLE]

where $\mathcal{A}_{i}$ is the set of pixels in the $i$-th superpixel, $\mathcal{B}_{i}$ is the corresponding 3D points, and $\left|\cdot\right|$ denotes the set size. Computation occurs only if $\left|\mathcal{B}_{i}\right|>0$. The feature distillation loss is calculated via InfoNCE [48]:

[TABLE]

where $\tau$ is the temperature. During distillation, the parameters of the image feature extractor backbone ${\theta^{\text{\tiny bck}}}$ are frozen, while other parameters, including ${\theta^{\text{\tiny hd}}}$, ${\omega_{h}}$, and ${\lambda}$, are updated.

In FSKD, the feature distillation loss is the mean square error between $\mathcal{G}^{\prime}$ and $\mathcal{F}$:

[TABLE]

The correspondence between coordinates in $\mathcal{G}^{\prime}$ and $\mathcal{F}$ is determined by extrinsic parameter calibration.

III-F Semantic-Based Knowledge Distillation

In FSKD, leveraging the full 2D semantic segmentation network $f_{\theta}$, we conduct semantic distillation by aligning the 3D network $g_{\omega}$ generated soft labels $\mathcal{T}^{C_{\text{cls}}\times N}$ with the output $\mathcal{S}^{C_{\text{cls}}\times H\times W}$ of $f_{\theta}$. The semantic distillation loss is computed using KL divergence:

[TABLE]

The overall knowledge distillation loss in FSKD, $\mathcal{L}_{\text{\tiny kd}}$, is a weighted sum of feature and semantic distillation losses: $\mathcal{L}_{\text{\tiny kd}}^{\text{\tiny FSKD}}=a\mathcal{L}_{\text{\tiny feat}}^{\text{\tiny FSKD}}+b\mathcal{L}_{\text{\tiny sem}}^{\text{\tiny FSKD}}$, where $a$ and $b$ are scaling factors. Due to the combined feature-based and semantic-based distillation, superpixels and the image projection head are unnecessary for feature alignment in FSKD.

IV Experiment

IV-A Datasets

Before FSKD, a complete 2D segmentation network $f_{\theta}$ is trained on the singlemodal nuImages dataset, resembling scenes in nuScenes but without overlapping data. nuImages annotations cover 26 classes, mapped to a commonly used set of 11 classes, a subset of nuScenes. nuScenes serves as the main target modality dataset $\mathbf{T}$. Following [6], 100 scenes are set aside for hyperparameter tuning, while all keyframes from the remaining 600 training scenes are used for crossmodal knowledge distillation. The original 32 classes in nuScenes are reduced to 16 by merging similar categories [6]. To validate generalization, we also fine-tune the 3D network $h_{\omega_{h}}$ on SemanticKITTI [49].

IV-B Few-Shot Semantic Segmentation

Few-shot semantic segmentation assesses various methods for pretraining the 3D feature extractor $h_{\omega_{h}}$. Since each of our methods supports this task, we include both in the evaluation. We compare our methods against three strong baselines, including the top-performing predecessor SLidR [6].

Training settings

In UDAKD, we directly use the ResNet-50 pretrained on ImageNet [50] with MoCo v2 [51] as the 2D teacher network’s backbone without fine-tuning. During knowledge distillation, we use the SGD optimizer with a momentum of 0.9, damping of 0.1, weight decay of 0.0001, and an initial learning rate of 0.5 to optimize the image projection head, 3D feature extractor, and domain adaptation module. Training lasts up to 50 epochs with a batch size of 4, employing the cosine annealing scheduler to decay the learning rate to 0 at the end of training. The temperature $\tau$ in (6) is set to 0.07. Each image is partitioned into a maximum of 150 SLIC superpixels.

Before conducting FSKD, $f_{\theta}$ is trained on the nuImages dataset for 100 epochs with cross-entropy and Lovász loss. During the distillation phase, we freeze $f_{\theta}$ and optimize the 3D feature extractor $h_{\omega_{h}}$ and the domain adaptation module $m_{\lambda}$ using SGD. The momentum is set to 0.9, damping to 0.1, weight decay to 0.0001, and the initial learning rate to 0.05 for $h_{\omega_{h}}$ and 5e-5 for $m_{\lambda}$, with a batch size of 16. Training is conducted for a maximum of 50 epochs using the cosine annealing scheduler. The ratio of $\mathcal{L}_{\text{\tiny feat}}$ to $\mathcal{L}_{\text{\tiny sem}}$ is set at 10:1.

Baselines

We compare our methods with I2P-MAE [32], PPKT [8], and SLidR [6]. I2P-MAE is a sophisticated masked autoencoding framework for image-to-point knowledge transfer. We include it to evaluate the adaptability of methods excelling in dense synthetic or indoor point cloud contexts to the challenges of sparse point cloud scenarios in autonomous driving. Our implementations of PPKT and SLidR are based on the publicly available code from [6].

Evaluation results

We evaluate the mIoU achieved by various methods on nuScenes and SemanticKITTI validation sets after fine-tuning on $1\%$ or $100\%$ annotated scenes from their respective training sets. In the fine-tuning phase, the 3D feature extractor parameters $\omega_{h}$, obtained through diverse methods, initialize $h_{\widetilde{\omega_{h}}}$. Simultaneously, a classifier $\widetilde{k}_{\widetilde{\omega_{k}}}$, trained from scratch, amalgamates to form the 3D segmentation network $\widetilde{g}_{\widetilde{\omega}}=\widetilde{k}_{\widetilde{\omega_{k}}}\circ h_{\widetilde{\omega_{h}}}$. Results are presented in Table I, where the Random baseline denotes random initialization of $h_{\widetilde{\omega_{h}}}$. Pretraining with I2P-MAE underperforms due to masking much of the already sparse point cloud. Our UDAKD approach extends SLidR by adding a domain adaptation module $m_{\lambda}$ with a 3-layer self-calibration convolution. Including $m_{\lambda}$ improves mIoU by 1 percentage point, demonstrating its effectiveness.

In FSKD, we utilize a 2D network trained on labeled image data, unlike other methods that use a frozen ResNet-50 pretrained via self-supervision. Although this approach might seem biased, the effectiveness of self-supervision relies on balanced, high-quality datasets [50, 52], often curated by human annotators. By training the image network on nuImages with 11 classes, FSKD achieves competitive performance across all 16 nuScenes classes after distillation, demonstrating its ability to enable the 3D network to extract generalizable features.

Visual inspection

We input 3D data from the nuScenes validation set into feature extractors $h_{\omega_{h}}$ pretrained with various methods to generate feature vectors. These vectors undergo t-SNE dimensionality reduction, with the resulting points color-coded by ground truth semantic labels, as shown in Fig. 2. Compared to SLidR [6], UDAKD shows clearer separation among dominant categories, such as the deep purple drivable surface, deep green manmade, orange terrain, and brown vegetation, with tighter intra-category clustering. FSKD, which utilizes 2D image labels during 2D network training, shows further improved 3D feature extraction performance.

IV-C Zero-Shot Semantic Segmentation

In the zero-shot segmentation setup of FSKD, the 2D classifier $d_{\theta_{d}}$ trained on images is reused as the 3D classifier $k_{\omega_{k}}$ with identical structure and parameters. Although not strictly necessary for zero-shot segmentation, this sharing minimally affects training effectiveness and is crucial for zero-shot domain adaptation, discussed in the next subsection. Other processes and hyperparameters remain consistent with those used in few-shot semantic segmentation.

Baselines

Training 3D networks without direct 3D supervision is largely unexplored. To establish baselines, we adapt OpenScene [36], 2DPASS [53], and MVF [54]. OpenScene uses cosine similarity loss to align 3D features with 2D-3D fusion features. To ensure comparability with other baselines, we train the 2D network with supervised data from the singlemodal dataset $\mathbf{S}$ in the source domain, instead of text-image pairs used in the original paper [36]. In MVF, the 2D network generates pseudo-labels, assigning each 3D point the corresponding 2D semantic label if aligned with a 2D pixel. For 2DPASS, lacking ground truth 3D labels, 2D knowledge is transferred to 3D in two steps: fusion features are generated using the original 2DPASS approach but limited to a single scale, and combined with 2D and 3D features as inputs to a fixed classifier trained on 2D images, which outputs semantic logit maps. The total loss includes KL divergence between the fused soft labels and both the 2D and 3D soft labels.

Evaluation results

We evaluate mIoU for the 11 common semantic classes in nuImages and nuScenes on the nuScenes validation set without fine-tuning the 3D network. As shown in Table II, these results are not directly comparable to the few-shot segmentation results in Table I. The cosine similarity used for feature alignment, as seen in OpenScene, proves less effective in our context. Our method outperforms the pseudo-label approach of MVF by 7.9 percentage points. While 2DPASS shows a slight improvement over MVF ($+1.0$ mIoU), it still trails our FSKD method by 6.9 mIoU points.

IV-D Further Insights into FSKD

Annotation efficiency

A key question is whether the performance of FSKD results from overfitting the 3D student to the 2D teacher’s outputs. To explore this, we fine-tune SLidR [6] and FSKD-pretrained 3D networks using different proportions of 3D labels from the nuScenes training set. We evaluate the mIoU on the validation set for the 11 semantic classes, and the results are summarized in Table III. As annotation proportions increase, both methods improve; however, even with $100\%$ annotation availability, the FSKD-pretrained network outperforms SLidR by 0.4 percentage points. This indicates that FSKD not only performs well in zero-shot learning but also enhances the 3D network’s capacity to extract generalizable features.

Zero-shot domain adaptation

Our approach enables zero-shot domain adaptation by sharing the same classifier between 3D and 2D networks. After knowledge distillation, the 2D network $f_{\theta}$ is retrained on nuImages without mapping its 25 semantic classes and 1 noise class to 11 semantic classes. The noise and ego-vehicle classes are treated as noise, using the remaining 24 classes without merging. During training, we freeze the 2D feature extractor $e_{\theta_{e}}$ and replace its classifier with a new one, $d_{\theta_{d}^{\prime}}^{\prime}$, forming SegNet*. We then link $d_{\theta_{d}^{\prime}}^{\prime}$ to the previously distilled 3D feature extractor $h_{\omega_{h}}$ and domain adaptation module $m_{\lambda}$, forming FSKD*, denoted as $g_{\omega^{\prime}}^{\prime}=d_{\theta_{d}^{\prime}}^{\prime}\circ m_{\lambda}\circ h_{\omega_{h}}$. With closely aligned 3D and 2D features during FSKD, $g_{\omega^{\prime}}^{\prime}$ can directly recognize new classes. Table IV shows the mIoU for FSKD* and SegNet* across various classes. The 2D network distinguishes debris, pushable/pullable objects, and bicycle racks (previously classified as noise), as well as adult pedestrians and construction workers (previously grouped as pedestrians). The 3D network, sharing the same classifier, differentiates these classes without any parameter adjustments or additional 3D annotations.

IV-E Ablation Study

UDAKD

In this study, we evaluate UDAKD by omitting the domain adaptation module $m_{\lambda}$ and superpixels. The results in Table V demonstrate that incorporating $m_{\lambda}$ not only boosts performance but also reduces dependency on superpixels. Even without superpixels, our UDAKD still outperforms SLidR [6], which considers the introduction of superpixels as its major contribution.

FSKD

The FSKD ablation study measures the mIoU of the 3D network trained on the nuScenes training set via crossmodal knowledge distillation, without additional fine-tuning using 3D ground truth labels. Table VI highlights the importance of each component in UDAKD. Removing semantic-based distillation reduces performance by 15.7 percentage points, underscoring the importance of aligning semantic soft labels for zero-shot crossmodal supervision. The domain adaptation module is essential for isolating modality-specific information and aligning 2D and 3D outputs at the feature level, enabling effective knowledge distillation at both feature and semantic levels.

Fig. 3 shows the feature map outputs of the 3D feature extractor $h_{\omega_{h}}$ with different components ablated. FSKD effectively distinguishes between classes and maintains consistency within the same class, clearly identifying roads, cars, and buildings. When components are removed, the features’ ability to convey semantic information diminishes. Even in the w/o soft labels scenario, there is significant risk of confusion, such as mistaking road surfaces color-coded in light blue for cars or erroneously associating cars color-coded in red and yellow with traffic lights.

V Conclusion

This work introduces two crossmodal knowledge distillation methods: Unsupervised Domain Adaptation Knowledge Distillation (UDAKD) and Feature and Semantic-based Knowledge Distillation (FSKD). UDAKD utilizes unlabeled image data for 2D teacher training, while FSKD leverages labeled images. Experimental results demonstrate the superiority of both UDAKD and FSKD over existing state-of-the-art methods.

While we highlight the role of a domain adaptation module in the 3D network, further research is needed to optimize its structure for crossmodal information transfer. Additionally, although 2D-3D correspondence can act as a form of supervision, we classify UDAKD as unsupervised due to the reliable, one-time extrinsic parameter calibration used to establish this correspondence. Accurately determining coordinate transformations between sparse point clouds and images without prior knowledge remains a substantive enigma.

ACKNOWLEDGMENT

The work is supported in part by the National Natural Science Foundation of China (No. 62176004, No. U1713217), Intelligent Robotics and Autonomous Vehicle Lab (RAV), the Fundamental Research Funds for the Central Universities, and High-performance Computing Platform of Peking University.