Learning From Biased Soft Labels

TL;DR

This paper investigates the effectiveness of biased soft labels in knowledge distillation, providing theoretical indicators and conditions that ensure their usefulness despite biases, supported by empirical validation.

Contribution

It introduces two comprehensive indicators to measure biased soft label effectiveness and establishes conditions for classifier consistency and learnability, extending understanding beyond unbiased labels.

Findings

Biased soft labels can effectively train classifiers.

Theoretical indicators predict when biased soft labels are beneficial.

Experimental validation confirms the theory's predictions.

Abstract

Knowledge distillation has been widely adopted in a variety of tasks and has achieved remarkable successes. Since its inception, many researchers have been intrigued by the dark knowledge hidden in the outputs of the teacher model. Recently, a study has demonstrated that knowledge distillation and label smoothing can be unified as learning from soft labels. Consequently, how to measure the effectiveness of the soft labels becomes an important question. Most existing theories have stringent constraints on the teacher model or data distribution, and many assumptions imply that the soft labels are close to the ground-truth labels. This paper studies whether biased soft labels are still effective. We present two more comprehensive indicators to measure the effectiveness of such soft labels. Based on the two indicators, we give sufficient conditions to ensure biased soft label based learners are classifier-consistent and ERM learnable. The theory is applied to three weakly-supervised frameworks. Experimental results validate that biased soft labels can also teach good students, which corroborates the soundness of the theory.