Self-Distillation Amplifies Regularization in Hilbert Space

TL;DR

This paper provides a theoretical analysis of self-distillation in deep learning, revealing how iterative self-distillation acts as a form of regularization in Hilbert space, affecting model complexity and performance.

Contribution

It introduces the first rigorous theoretical framework for understanding self-distillation, showing its role in modifying regularization and basis function selection.

Findings

Self-distillation limits the number of basis functions in the model.

Few rounds of self-distillation reduce overfitting.

Too many rounds can cause underfitting and degrade performance.

Abstract

Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another. In particular, when the architectures are identical, this is called self-distillation. The idea is to feed in predictions of the trained model as new target values for retraining (and iterate this loop possibly a few times). It has been empirically observed that the self-distilled model often achieves higher accuracy on held out data. Why this happens, however, has been a mystery: the self-distillation dynamics does not receive any new information about the task and solely evolves by looping over training. To the best of our knowledge, there is no rigorous understanding of this phenomenon. This work provides the first theoretical analysis of self-distillation. We focus on fitting a nonlinear function to training data, where the model space is Hilbert space and fitting is subject to $\ell_2$ regularization in this function space. We show that self-distillation iterations modify regularization by progressively limiting the number of basis functions that can be used to represent the solution. This implies (as we also verify empirically) that while a few rounds of self-distillation may reduce over-fitting, further rounds may lead to under-fitting and thus worse performance.