Knowledge Distillation Must Account for What It Loses

TL;DR

This paper emphasizes that knowledge distillation should evaluate not only retained task performance but also the preservation of teacher capabilities to ensure reliability.

Contribution

It introduces the concept of accounting for what is lost during distillation, proposing a taxonomy and a reporting framework for accountable distillation.

Findings

Current evaluation often overlooks capability losses.

Losses in teacher capabilities are measurable and recurring.

Proposed framework improves transparency in distillation outcomes.

Abstract

This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large teacher models into deployable students, yet headline metrics can obscure losses in the capabilities that make teacher behavior reliable. Conceptually, we show that current evaluation often assumes retained task scores imply retained teacher capabilities. Reframing distillation as a lossy projection exposes this flaw: students may match selected teacher observables without preserving the capabilities that make them reliable. We then synthesize existing evidence into a taxonomy of off-metric distillation losses, showing that such losses are concrete, recurring, and measurable, yet often unaccounted for when studies report what students retain rather than what they lose. To make the position actionable, we propose scenario-specific preservation targets and a Distillation Loss Statement that reports what was preserved, what was lost, and why the remaining losses are acceptable. The goal is not lossless distillation, but accountable distillation.