Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting

TL;DR

This paper analyzes the effects of self-training in overparameterized linear regression, revealing a trade-off between signal forgetting and denoising that impacts test risk and optimal stopping, with theoretical and empirical validation.

Contribution

It provides a theoretical framework for understanding self-training dynamics in linear models, including risk recursions, spectral filtering effects, and a data-driven stopping criterion.

Findings

Test risk exhibits a U-shaped curve over iterations.

Iteration acts as a spectral filter preserving strong eigendirections.

Proposed cross-validation method accurately estimates optimal stopping time.

Abstract

Iterative self-training (self-distillation) repeatedly refits a model on pseudo-labels generated by its own predictions. We study this procedure in overparameterized linear regression: an initial estimator is trained on noisy labels, and each subsequent iterate is trained on fresh covariates with noiseless pseudo-labels from the previous model. In the high-dimensional regime, we derive deterministic-equivalent recursions for the prediction risk and effective noise across iterations, and prove that the empirical quantities concentrate sharply around these limits. The recursion separates two competing forces: a systematic component that grows with iteration due to progressive signal forgetting, and a stochastic component that decays due to denoising via repeated data-dependent projections. Their interaction yields a $U$-shaped test-risk curve and an optimal early-stopping time. In spiked covariance models, iteration further acts as an iteration-dependent spectral filter that preserves strong eigendirections while suppressing weaker ones, inducing an implicit form of soft feature selection distinct from ridge regression. Finally, we propose an iterated generalized cross-validation criterion and prove its uniform consistency for estimating the risk along the self-training trajectory, enabling fully data-driven selection of the stopping time and regularization. Experiments on synthetic covariances validate the theory and illustrate the predicted denoising-forgetting trade-off.