SLaM: Student-Label Mixing for Distillation with Unlabeled Examples

TL;DR

SLaM introduces a novel student-label mixing approach for knowledge distillation with unlabeled data, improving student model performance by reducing pseudo-label noise and providing theoretical guarantees.

Contribution

The paper proposes Student-Label Mixing (SLaM), a new method that enhances knowledge distillation with unlabeled data and offers theoretical analysis and improved sample complexity.

Findings

SLaM outperforms prior methods on standard benchmarks.

Provides theoretical guarantees for the distillation process.

Improves sample complexity for learning halfspaces with margin.

Abstract

Knowledge distillation with unlabeled examples is a powerful training paradigm for generating compact and lightweight student models in applications where the amount of labeled data is limited but one has access to a large pool of unlabeled data. In this setting, a large teacher model generates ``soft'' pseudo-labels for the unlabeled dataset which are then used for training the student model. Despite its success in a wide variety of applications, a shortcoming of this approach is that the teacher's pseudo-labels are often noisy, leading to impaired student performance. In this paper, we present a principled method for knowledge distillation with unlabeled examples that we call Student-Label Mixing (SLaM) and we show that it consistently improves over prior approaches by evaluating it on several standard benchmarks. Finally, we show that SLaM comes with theoretical guarantees; along the way we give an algorithm improving the best-known sample complexity for learning halfspaces with margin under random classification noise, and provide the first convergence analysis for so-called ``forward loss-adjustment" methods.