Logit-Based Losses Limit the Effectiveness of Feature Knowledge Distillation

TL;DR

This paper introduces a feature-only knowledge distillation framework that surpasses traditional logit-based methods by leveraging latent representations and a new knowledge quality metric, achieving significant accuracy improvements.

Contribution

The authors propose a novel feature-based knowledge distillation method that excludes logit-based losses and uses a new metric to identify effective teacher layers, leading to state-of-the-art results.

Findings

Achieves up to 15% top-1 accuracy boost over standard methods.

Effective across diverse datasets and model architectures.

Introduces a knowledge quality metric for layer selection.

Abstract

Knowledge distillation (KD) methods can transfer knowledge of a parameter-heavy teacher model to a light-weight student model. The status quo for feature KD methods is to utilize loss functions based on logits (i.e., pre-softmax class scores) and intermediate layer features (i.e., latent representations). Unlike previous approaches, we propose a feature KD framework for training the student's backbone using feature-based losses exclusively (i.e., without logit-based losses such as cross entropy). Leveraging recent discoveries about the geometry of latent representations, we introduce a knowledge quality metric for identifying which teacher layers provide the most effective knowledge for distillation. Experiments on three image classification datasets with four diverse student-teacher pairs, spanning convolutional neural networks and vision transformers, demonstrate our KD method achieves state-of-the-art performance, delivering top-1 accuracy boosts of up to 15% over standard approaches. We publically share our code to facilitate future work at https://github.com/Thegolfingocto/KD_wo_CE.