Boosting of Head Pose Estimation by Knowledge Distillation

TL;DR

This paper introduces a novel response-based knowledge distillation method that enhances head pose estimation accuracy, allowing student models to outperform teacher models and improving state-of-the-art results on multiple datasets.

Contribution

The paper presents a two-stage KD approach that boosts head pose estimation models, enabling deeper networks to be trained more effectively and partially surpassing existing state-of-the-art performance.

Findings

KD improves head pose estimation accuracy by 7.7% on average.

Student models outperform teacher models in this KD framework.

The method achieves state-of-the-art results on multiple datasets.

Abstract

We propose a response-based method of knowledge distillation (KD) for the head pose estimation problem. A student model trained by the proposed KD achieves results better than a teacher model, which is atypical for the response-based method. Our method consists of two stages. In the first stage, we trained the base neural network (NN), which has one regression head and four regression via classification (RvC) heads. We build the convolutional ensemble over the base NN using offsets of face bounding boxes over a regular grid. In the second stage, we perform KD from the convolutional ensemble into the final NN with one RvC head. The KD improves the results by an average of 7.7\% compared to base NN. This feature makes it possible to use KD as a booster and effectively train deeper NNs. NNs trained by our KD method partially improved the state-of-the-art results. KD-ResNet152 has the best results, and KD-ResNet18 has a better result on the AFLW2000 dataset than any previous method.We have made publicly available trained NNs and face bounding boxes for the 300W-LP, AFLW, AFLW2000, and BIWI datasets.Our method potentially can be effective for other regression problems.