Feature Adversarial Distillation for Point Cloud Classification

TL;DR

This paper introduces Feature Adversarial Distillation (FAD), a novel method for improving knowledge transfer in point cloud classification by using adversarial training to reduce information loss during model compression.

Contribution

It proposes a generic adversarial loss function for point cloud distillation that enhances knowledge transfer and reduces information loss in model compression.

Findings

Reduced information loss by 40x model compression

Maintained competitive classification performance

Effective on ModelNet40 and ScanObjectNN datasets

Abstract

Due to the point cloud's irregular and unordered geometry structure, conventional knowledge distillation technology lost a lot of information when directly used on point cloud tasks. In this paper, we propose Feature Adversarial Distillation (FAD) method, a generic adversarial loss function in point cloud distillation, to reduce loss during knowledge transfer. In the feature extraction stage, the features extracted by the teacher are used as the discriminator, and the students continuously generate new features in the training stage. The feature of the student is obtained by attacking the feedback from the teacher and getting a score to judge whether the student has learned the knowledge well or not. In experiments on standard point cloud classification on ModelNet40 and ScanObjectNN datasets, our method reduced the information loss of knowledge transfer in distillation in 40x model compression while maintaining competitive performance.