The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model

TL;DR

This paper investigates how self-distillation improves binary classification on noisy data, revealing denoising as the key factor and proposing heuristics like early stopping and bias fixing, validated through theoretical analysis and experiments.

Contribution

It provides a theoretical analysis of self-distillation's effectiveness in noisy Gaussian mixture models and introduces practical heuristics to enhance its performance.

Findings

Denoising via hard pseudo-labels is the main driver of SD's effectiveness.

Early stopping of stages broadly improves SD performance.

Bias parameter fixing aids in label imbalance scenarios.

Abstract

Self-distillation (SD), a technique where a model improves itself using its own predictions, has attracted attention as a simple yet powerful approach in machine learning. Despite its widespread use, the mechanisms underlying its effectiveness remain unclear. In this study, we investigate the efficacy of hyperparameter-tuned multi-stage SD with a linear classifier for binary classification on noisy Gaussian mixture data. For the analysis, we employ the replica method from statistical physics. Our findings reveal that the primary driver of SD's performance improvement is denoising through hard pseudo-labels, with the most notable gains observed in moderately sized datasets. We also identify two practical heuristics to enhance SD: early stopping that limits the number of stages, which is broadly effective, and bias parameter fixing, which helps under label imbalance. To empirically validate our theoretical findings derived from our toy model, we conduct additional experiments on CIFAR-10 classification using pretrained ResNet backbone. These results provide both theoretical and practical insights, advancing our understanding and application of SD in noisy settings.