Visualizing the embedding space to explain the effect of knowledge distillation

TL;DR

This paper uses visualization techniques to analyze how knowledge distillation leads to more compact and effective representation spaces in neural networks, explaining its success in improving model performance.

Contribution

It introduces a visualization approach using t-SNE and IVIS to understand the effects of knowledge distillation on network representations.

Findings

Distillation results in more compact representation spaces.

Higher accuracy correlates with earlier layer compression.

Visualizations reveal how distillation guides network learning.

Abstract

Recent research has found that knowledge distillation can be effective in reducing the size of a network and in increasing generalization. A pre-trained, large teacher network, for example, was shown to be able to bootstrap a student model that eventually outperforms the teacher in a limited label environment. Despite these advances, it still is relatively unclear \emph{why} this method works, that is, what the resulting student model does 'better'. To address this issue, here, we utilize two non-linear, low-dimensional embedding methods (t-SNE and IVIS) to visualize representation spaces of different layers in a network. We perform a set of extensive experiments with different architecture parameters and distillation methods. The resulting visualizations and metrics clearly show that distillation guides the network to find a more compact representation space for higher accuracy already in earlier layers compared to its non-distilled version.