Efficient Point Cloud Classification via Offline Distillation Framework and Negative-Weight Self-Distillation Technique

TL;DR

This paper presents an efficient offline distillation framework with a negative-weight self-distillation technique for point cloud classification, reducing computational resources while maintaining high accuracy in resource-constrained environments.

Contribution

It introduces an offline recording strategy and negative-weight self-distillation to improve model efficiency and performance in point cloud classification.

Findings

Achieves state-of-the-art accuracy with fewer parameters

Reduces hardware demands during training

Balances performance and model complexity effectively

Abstract

The rapid advancement in point cloud processing technologies has significantly increased the demand for efficient and compact models that achieve high-accuracy classification. Knowledge distillation has emerged as a potent model compression technique. However, traditional KD often requires extensive computational resources for forward inference of large teacher models, thereby reducing training efficiency for student models and increasing resource demands. To address these challenges, we introduce an innovative offline recording strategy that avoids the simultaneous loading of both teacher and student models, thereby reducing hardware demands. This approach feeds a multitude of augmented samples into the teacher model, recording both the data augmentation parameters and the corresponding logit outputs. By applying shape-level augmentation operations such as random scaling and translation, while excluding point-level operations like random jittering, the size of the records is significantly reduced. Additionally, to mitigate the issue of small student model over-imitating the teacher model's outputs and converging to suboptimal solutions, we incorporate a negative-weight self-distillation strategy. Experimental results demonstrate that the proposed distillation strategy enables the student model to achieve performance comparable to state-of-the-art models while maintaining lower parameter count. This approach strikes an optimal balance between performance and complexity. This study highlights the potential of our method to optimize knowledge distillation for point cloud classification tasks, particularly in resource-constrained environments, providing a novel solution for efficient point cloud analysis.