Visualizing the embedding space to explain the effect of knowledge distillation
Hyun Seung Lee, Christian Wallraven

TL;DR
This paper uses visualization techniques to analyze how knowledge distillation leads to more compact and effective representation spaces in neural networks, explaining its success in improving model performance.
Contribution
It introduces a visualization approach using t-SNE and IVIS to understand the effects of knowledge distillation on network representations.
Findings
Distillation results in more compact representation spaces.
Higher accuracy correlates with earlier layer compression.
Visualizations reveal how distillation guides network learning.
Abstract
Recent research has found that knowledge distillation can be effective in reducing the size of a network and in increasing generalization. A pre-trained, large teacher network, for example, was shown to be able to bootstrap a student model that eventually outperforms the teacher in a limited label environment. Despite these advances, it still is relatively unclear \emph{why} this method works, that is, what the resulting student model does 'better'. To address this issue, here, we utilize two non-linear, low-dimensional embedding methods (t-SNE and IVIS) to visualize representation spaces of different layers in a network. We perform a set of extensive experiments with different architecture parameters and distillation methods. The resulting visualizations and metrics clearly show that distillation guides the network to find a more compact representation space for higher accuracy already…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Visualization and Analytics · Neural Networks and Applications · Computational Physics and Python Applications
MethodsKnowledge Distillation
