Knowledge Distillation Based on Transformed Teacher Matching

TL;DR

This paper introduces transformed teacher matching (TTM), a novel knowledge distillation method that omits student-side temperature scaling, leveraging inherent regularization for improved generalization, and further enhances it with weighted TTM (WTTM) for state-of-the-art results.

Contribution

The paper proposes TTM, a new KD variant that removes student temperature scaling and incorporates Rènyi entropy regularization, along with WTTM, an adaptive weighting scheme for better performance.

Findings

TTM outperforms traditional KD in generalization.

WTTM achieves state-of-the-art accuracy.

Inherent regularization improves student model performance.

Abstract

As a technique to bridge logit matching and probability distribution matching, temperature scaling plays a pivotal role in knowledge distillation (KD). Conventionally, temperature scaling is applied to both teacher's logits and student's logits in KD. Motivated by some recent works, in this paper, we drop instead temperature scaling on the student side, and systematically study the resulting variant of KD, dubbed transformed teacher matching (TTM). By reinterpreting temperature scaling as a power transform of probability distribution, we show that in comparison with the original KD, TTM has an inherent R\'enyi entropy term in its objective function, which serves as an extra regularization term. Extensive experiment results demonstrate that thanks to this inherent regularization, TTM leads to trained students with better generalization than the original KD. To further enhance student's capability to match teacher's power transformed probability distribution, we introduce a sample-adaptive weighting coefficient into TTM, yielding a novel distillation approach dubbed weighted TTM (WTTM). It is shown, by comprehensive experiments, that although WTTM is simple, it is effective, improves upon TTM, and achieves state-of-the-art accuracy performance. Our source code is available at https://github.com/zkxufo/TTM.