SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillation

TL;DR

This paper introduces SRKD, a knowledge distillation framework that efficiently transfers geometric and semantic knowledge from large transformer models to smaller ones for 3D point cloud segmentation, improving performance and deployment feasibility.

Contribution

The paper proposes a novel structure- and relation-aware knowledge distillation method with relation alignment and cross-sample strategies for efficient 3D point cloud segmentation.

Findings

Achieves state-of-the-art performance with smaller models

Enhances structural understanding through relation alignment

Improves generalization across diverse point clouds

Abstract

3D point cloud segmentation faces practical challenges due to the computational complexity and deployment limitations of large-scale transformer-based models. To address this, we propose a novel Structure- and Relation-aware Knowledge Distillation framework, named SRKD, that transfers rich geometric and semantic knowledge from a large frozen teacher model (>100M) to a lightweight student model (<15M). Specifically, we propose an affinity matrix-based relation alignment module, which distills structural dependencies from the teacher to the student through point-wise similarity matching, enhancing the student's capability to learn contextual interactions. Meanwhile, we introduce a cross-sample mini-batch construction strategy that enables the student to perceive stable and generalized geometric structure. This aligns across diverse point cloud instances of the teacher, rather than within a single sample. Additionally, KL divergence is applied to align semantic distributions, and ground-truth supervision further reinforces accurate segmentation. Our method achieves state of the art performance with significantly reduced model complexity, demonstrating its effectiveness and efficiency in real-world deployment scenarios. Our Code is available at https://github.com/itsnotacie/SRKD.