Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models

TL;DR

This paper establishes that s-step self-distillation is statistically optimal among spectral shrinkage estimators for spiked covariance models, explaining its effectiveness in improving model performance.

Contribution

It provides the first rigorous statistical analysis of self-distillation, showing its optimality among spectral shrinkage estimators and connecting it with classical shrinkage methods.

Findings

s-step self-distillation achieves optimal performance among spectral shrinkage estimators

Any fewer than s steps results in suboptimal estimators

Optimal Ridge regression outperforms other spectral shrinkage estimators in isotropic cases

Abstract

Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in spiked covariance models, by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We establish that for spiked covariance matrices with $s$ spikes, $s$-step self-distillation achieves optimal performance among spectral shrinkage estimators, outperforming well-known estimators in statistics and machine learning. Moreover, we show that $s$ steps are necessary for optimality: any $(s-k)$-step distilled estimator is strictly suboptimal for $1 \leq k \leq s$. For the special subclass of isotropic covariances, we show that optimally tuned Ridge regression performs best among spectral shrinkage estimators. We also study a federated approach where multiple data centers share spectral shrinkage estimators and a common server seeks to aggregate them to achieve optimal performance. In this case, we find that the best local rule again takes the form of self-distillation, though it differs from the optimal rule when data are hosted centrally on a single server. Together, our results elucidate why self-distillation improves predictive performance and provide a broader statistical framework connecting it with classical shrinkage-based methods.