Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers

TL;DR

This paper analyzes why feature-based knowledge distillation fails for Vision Transformers by revealing a fundamental representational mismatch, and proposes insights for designing better ViT compression methods.

Contribution

It introduces a novel 'distillation dynamics' framework to analyze ViT feature distillation failure and identifies the core representational mismatch as the root cause.

Findings

ViTs exhibit a U-shaped information processing pattern.

High-dimensional encoding in teacher models causes transfer issues.

Naive feature mimicry harms student performance.

Abstract

While feature-based knowledge distillation has proven highly effective for compressing CNNs, these techniques unexpectedly fail when applied to Vision Transformers (ViTs), often performing worse than simple logit-based distillation. We provide the first comprehensive analysis of this phenomenon through a novel analytical framework termed as "distillation dynamics", combining frequency spectrum analysis, information entropy metrics, and activation magnitude tracking. Our investigation reveals that ViTs exhibit a distinctive U-shaped information processing pattern: initial compression followed by expansion. We identify the root cause of negative transfer in feature distillation: a fundamental representational paradigm mismatch between teacher and student models. Through frequency-domain analysis, we show that teacher models employ distributed, high-dimensional encoding strategies in later layers that smaller student models cannot replicate due to limited channel capacity. This mismatch causes late-layer feature alignment to actively harm student performance. Our findings reveal that successful knowledge transfer in ViTs requires moving beyond naive feature mimicry to methods that respect these fundamental representational constraints, providing essential theoretical guidance for designing effective ViTs compression strategies. All source code and experimental logs are provided at https://github.com/thy960112/Distillation-Dynamics.