Self Regulated Learning Mechanism for Data Efficient Knowledge Distillation

TL;DR

This paper introduces a significance-based self-regulated learning approach for data-efficient knowledge distillation, enabling models to achieve competitive performance using fewer training samples by focusing on significant data points.

Contribution

The work proposes a novel self-regulation mechanism that restricts sample participation based on significance, improving data efficiency in knowledge distillation.

Findings

Achieves similar performance to state-of-the-art methods with fewer samples

Uses significance measures to weigh sample contributions during distillation

Demonstrates effectiveness on benchmark datasets

Abstract

Existing methods for distillation do not efficiently utilize the training data. This work presents a novel approach to perform distillation using only a subset of the training data, making it more data-efficient. For this purpose, the training of the teacher model is modified to include self-regulation wherein a sample in the training set is used for updating model parameters in the backward pass either if it is misclassified or the model is not confident enough in its prediction. This modification restricts the participation of samples, unlike the conventional training method. The number of times a sample participates in the self-regulated training process is a measure of its significance towards the model's knowledge. The significance values are used to weigh the losses incurred on the corresponding samples in the distillation process. This method is named significance-based distillation. Two other methods are proposed for comparison where the student model learns by distillation and incorporating self-regulation as the teacher model, either utilizing the significance information computed during the teacher's training or not. These methods are named hybrid and regulated distillations, respectively. Experiments on benchmark datasets show that the proposed methods achieve similar performance as other state-of-the-art methods for knowledge distillation while utilizing a significantly less number of samples.