An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillation

TL;DR

This paper introduces a novel knowledge distillation method tailored for regression tasks, utilizing a new loss function and a multi-task network to improve small model training on noisy data.

Contribution

It proposes a formalism for regression knowledge distillation, including a teacher outlier rejection loss and a multi-task network architecture for better noise handling.

Findings

Consistent accuracy improvements across datasets.

Effective noise handling in training labels.

Enhanced student model performance.

Abstract

Compressing deep neural network (DNN) models becomes a very important and necessary technique for real-world applications, such as deploying those models on mobile devices. Knowledge distillation is one of the most popular methods for model compression, and many studies have been made on developing this technique. However, those studies mainly focused on classification problems, and very few attempts have been made on regression problems, although there are many application of DNNs on regression problems. In this paper, we propose a new formalism of knowledge distillation for regression problems. First, we propose a new loss function, teacher outlier rejection loss, which rejects outliers in training samples using teacher model predictions. Second, we consider a multi-task network with two outputs: one estimates training labels which is in general contaminated by noisy labels; And the other estimates teacher model's output which is expected to modify the noise labels following the memorization effects. By considering the multi-task network, training of the feature extraction of student models becomes more effective, and it allows us to obtain a better student model than one trained from scratch. We performed comprehensive evaluation with one simple toy model: sinusoidal function, and two open datasets: MPIIGaze, and Multi-PIE. Our results show consistent improvement in accuracy regardless of the annotation error level in the datasets.