Representation Disparity-aware Distillation for 3D Object Detection

TL;DR

This paper introduces a novel knowledge distillation method for 3D object detection that addresses representation disparity, significantly improving compact detector performance and surpassing teacher models in some cases.

Contribution

We propose a representation disparity-aware distillation (RDD) method based on information bottleneck to enhance knowledge transfer in 3D detectors with large disparity.

Findings

RDD outperforms existing KD methods in 3D detection tasks.

RDD achieves 57.1% mAP on nuScenes, surpassing teacher performance.

RDD reduces FLOPs while maintaining high detection accuracy.

Abstract

In this paper, we focus on developing knowledge distillation (KD) for compact 3D detectors. We observe that off-the-shelf KD methods manifest their efficacy only when the teacher model and student counterpart share similar intermediate feature representations. This might explain why they are less effective in building extreme-compact 3D detectors where significant representation disparity arises due primarily to the intrinsic sparsity and irregularity in 3D point clouds. This paper presents a novel representation disparity-aware distillation (RDD) method to address the representation disparity issue and reduce performance gap between compact students and over-parameterized teachers. This is accomplished by building our RDD from an innovative perspective of information bottleneck (IB), which can effectively minimize the disparity of proposal region pairs from student and teacher in features and logits. Extensive experiments are performed to demonstrate the superiority of our RDD over existing KD methods. For example, our RDD increases mAP of CP-Voxel-S to 57.1% on nuScenes dataset, which even surpasses teacher performance while taking up only 42% FLOPs.