Reducing Capacity Gap in Knowledge Distillation with Review Mechanism for Crowd Counting

TL;DR

This paper introduces ReviewKD, a novel review mechanism for knowledge distillation in crowd counting, which reduces the capacity gap and enhances performance beyond the teacher network.

Contribution

The paper proposes a review mechanism in KD models that refines density map estimates, effectively alleviating the capacity gap and improving lightweight crowd counting models.

Findings

ReviewKD outperforms existing lightweight models on six benchmarks.

The review mechanism can boost heavy models without architecture changes.

ReviewKD can surpass the performance of the teacher network.

Abstract

The lightweight crowd counting models, in particular knowledge distillation (KD) based models, have attracted rising attention in recent years due to their superiority on computational efficiency and hardware requirement. However, existing KD based models usually suffer from the capacity gap issue, resulting in the performance of the student network being limited by the teacher network. In this paper, we address this issue by introducing a novel review mechanism following KD models, motivated by the review mechanism of human-beings during the study. Thus, the proposed model is dubbed ReviewKD. The proposed model consists of an instruction phase and a review phase, where we firstly exploit a well-trained heavy teacher network to transfer its latent feature to a lightweight student network in the instruction phase, then in the review phase yield a refined estimate of the density map based on the learned feature through a review mechanism. The effectiveness of ReviewKD is demonstrated by a set of experiments over six benchmark datasets via comparing to the state-of-the-art models. Numerical results show that ReviewKD outperforms existing lightweight models for crowd counting, and can effectively alleviate the capacity gap issue, and particularly has the performance beyond the teacher network. Besides the lightweight models, we also show that the suggested review mechanism can be used as a plug-and-play module to further boost the performance of a kind of heavy crowd counting models without modifying the neural network architecture and introducing any additional model parameter.