Dynamic Temperature Scheduler for Knowledge Distillation

TL;DR

This paper introduces a novel dynamic temperature scheduler for knowledge distillation that adapts the temperature parameter during training based on the divergence between teacher and student, improving performance across vision and NLP tasks.

Contribution

The paper proposes the first temperature scheduling method that adjusts based on teacher-student divergence, enhancing knowledge distillation effectiveness.

Findings

DTS outperforms static-temperature baselines on multiple datasets.

Dynamic adjustment of temperature improves student model training.

Method is effective across vision and NLP tasks.

Abstract

Knowledge Distillation (KD) trains a smaller student model using a large, pre-trained teacher model, with temperature as a key hyperparameter controlling the softness of output probabilities. Traditional methods use a fixed temperature throughout training, which is suboptimal. Moreover, architectural differences between teacher and student often result in mismatched logit magnitudes. We demonstrate that students benefit from softer probabilities early in training but require sharper probabilities in later stages. We introduce Dynamic Temperature Scheduler (DTS), which adjusts temperature dynamically based on the cross-entropy loss gap between teacher and student. To our knowledge, this is the first temperature scheduling method that adapts based on the divergence between teacher and student distributions. Our method integrates seamlessly with existing KD frameworks. We validate DTS across multiple KD strategies on vision (CIFAR-100, Tiny-ImageNet) and NLP tasks (GLUE, Dolly, SelfIns, UnNI, S-NI), consistently outperforming static-temperature baselines. Code is available at https://github.com/Sibgat-Ul/DTS.