Sparse Knowledge Distillation: A Mathematical Framework for Probability-Domain Temperature Scaling and Multi-Stage Compression

TL;DR

This paper introduces a comprehensive mathematical framework for sparse knowledge distillation using probability-domain operators, providing theoretical insights into multi-stage pruning, convergence, and operator equivalences.

Contribution

It offers an operator-level analytical framework for sparse knowledge distillation, including bias-variance analysis, multi-stage pruning theory, convergence guarantees, and operator classification.

Findings

Bias-variance decompositions explain when sparse students outperform dense teachers.

Multi-stage pruning success is explained by a homotopy path in function space.

Convergence guarantees of O(1/n) rates for n-stage distillation are established.

Abstract

We develop a unified theoretical framework for sparse knowledge distillation based on probability-domain softening operators. While the equivalence $p^{1/T} \propto \mathrm{softmax}(z/T)$ is well known, our contribution is an operator-level analytical framework built on this foundation rather than the equivalence itself. The framework comprises four core components: (i) operator-agnostic bias--variance decompositions that characterize when sparse students outperform dense teachers, (ii) a homotopy path formalization of multi-stage pruning in function space explaining why iterative compression succeeds where one-shot pruning fails, (iii) convergence guarantees establishing $O(1/n)$ rates for $n$-stage distillation with explicit parameter dependence, and (iv) equivalence class characterizations identifying distinct probability-domain operators that yield identical student models under capacity constraints. We introduce an axiomatic definition of probability-domain softening operators based on ranking preservation, continuity, entropy monotonicity, identity, and boundary behavior, and show that multiple non-equivalent operator families satisfy these axioms. All learning-theoretic guarantees are shown to hold uniformly across this operator class, independent of implementation details. These results provide theoretical grounding for black-box teacher distillation, partial-access settings such as top-$k$ truncation and text-only outputs, and privacy-preserving model compression.