Hard Gate Knowledge Distillation -- Leverage Calibration for Robust and Reliable Language Model

TL;DR

This paper introduces a novel hard gate knowledge distillation method that leverages model calibration to determine when to distill knowledge from a teacher, improving language model robustness and calibration.

Contribution

It proposes a new gating mechanism based on calibration to decide when to distill knowledge, enhancing model generalization and reliability.

Findings

Improves language model calibration accuracy.

Enhances generalization in natural language generation.

Reduces miscalibration errors significantly.

Abstract

In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution. Knowledge of a teacher is considered a subject that holds inter-class relations which send a meaningful supervision to a student; hence, much effort has been put to find such knowledge to be distilled. In this paper, we explore a question that has been given little attention: "when to distill such knowledge." The question is answered in our work with the concept of model calibration; we view a teacher model not only as a source of knowledge but also as a gauge to detect miscalibration of a student. This simple and yet novel view leads to a hard gate knowledge distillation scheme that switches between learning from a teacher model and training data. We verify the gating mechanism in the context of natural language generation at both the token-level and the sentence-level. Empirical comparisons with strong baselines show that hard gate knowledge distillation not only improves model generalization, but also significantly lowers model calibration error.