On the Generalization vs Fidelity Paradox in Knowledge Distillation

TL;DR

This paper provides a large-scale empirical analysis of knowledge distillation in language models, revealing its benefits, limitations, and the complex relationship between performance and reasoning fidelity across different model sizes.

Contribution

It is the first comprehensive study examining how knowledge distillation affects various language models and reasoning tasks, highlighting key factors influencing its effectiveness.

Findings

KD improves smaller models' performance by up to 10%

Teacher performance has minimal impact on student outcomes

KD benefits diminish as model size increases

Abstract

Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. Despite the recent traction of KD research, its effectiveness for smaller language models (LMs) and the mechanisms driving knowledge transfer remain underexplored. In this work, we present the first large-scale empirical and statistical analysis of KD across models ranging from 0.5B to 7B parameters on 14 complex reasoning tasks in a zero-shot setting. Our findings reveal that KD can improve the average performance of smaller models by up to $10\%$, with a peak task specific gain of $22\%$, while providing only marginal benefits ($\sim 1.3\%$) for larger models. Surprisingly, teacher performance has a minimal impact on student outcomes, while teacher task expertise impacts KD effectiveness. A correlation study indicates that smaller LMs benefit more from KD, whereas larger LMs show diminished gains. Additionally, we uncover a misalignment between improvements in student performance and reasoning fidelity, suggesting that while KD enhances accuracy, it does not always maintain the structured decision-making processes of the teacher. Our ablation study further highlights the importance of teacher signals and logit smoothing in influencing students' performance after distillation. Overall, our study offers a comprehensive empirical and statistical assessment of KD, highlighting both its benefits and trade-offs when distilling knowledge from larger to smaller LMs.