Self-Distillation as Instance-Specific Label Smoothing

TL;DR

This paper explores how multi-generational self-distillation improves model generalization through increased prediction diversity, offering a new interpretation as instance-specific regularization and proposing a novel label smoothing method that enhances performance.

Contribution

It introduces a theoretical framework linking self-distillation to label smoothing via predictive diversity and presents a new instance-specific label smoothing technique that outperforms traditional methods.

Findings

Self-distillation increases prediction diversity, improving generalization.

Theoretical link between self-distillation and label smoothing.

Proposed method often outperforms classical label smoothing.

Abstract

It has been recently demonstrated that multi-generational self-distillation can improve generalization. Despite this intriguing observation, reasons for the enhancement remain poorly understood. In this paper, we first demonstrate experimentally that the improved performance of multi-generational self-distillation is in part associated with the increasing diversity in teacher predictions. With this in mind, we offer a new interpretation for teacher-student training as amortized MAP estimation, such that teacher predictions enable instance-specific regularization. Our framework allows us to theoretically relate self-distillation to label smoothing, a commonly used technique that regularizes predictive uncertainty, and suggests the importance of predictive diversity in addition to predictive uncertainty. We present experimental results using multiple datasets and neural network architectures that, overall, demonstrate the utility of predictive diversity. Finally, we propose a novel instance-specific label smoothing technique that promotes predictive diversity without the need for a separately trained teacher model. We provide an empirical evaluation of the proposed method, which, we find, often outperforms classical label smoothing.