itKD: Interchange Transfer-based Knowledge Distillation for 3D Object Detection

TL;DR

This paper introduces a novel knowledge distillation framework for 3D object detection that enhances lightweight models' accuracy and efficiency by using interchange transfer and attention mechanisms, validated on public datasets.

Contribution

The paper proposes an autoencoder-style knowledge distillation method with interchange transfer and attention loss for improved 3D detection in point clouds.

Findings

Improves lightweight 3D detection accuracy on Waymo and nuScenes datasets.

Reduces computational complexity while maintaining high detection performance.

Demonstrates superiority over existing methods in 3D object detection tasks.

Abstract

Point-cloud based 3D object detectors recently have achieved remarkable progress. However, most studies are limited to the development of network architectures for improving only their accuracy without consideration of the computational efficiency. In this paper, we first propose an autoencoder-style framework comprising channel-wise compression and decompression via interchange transfer-based knowledge distillation. To learn the map-view feature of a teacher network, the features from teacher and student networks are independently passed through the shared autoencoder; here, we use a compressed representation loss that binds the channel-wised compression knowledge from both student and teacher networks as a kind of regularization. The decompressed features are transferred in opposite directions to reduce the gap in the interchange reconstructions. Lastly, we present an head attention loss to match the 3D object detection information drawn by the multi-head self-attention mechanism. Through extensive experiments, we verify that our method can train the lightweight model that is well-aligned with the 3D point cloud detection task and we demonstrate its superiority using the well-known public datasets; e.g., Waymo and nuScenes.