In Teacher We Trust: Learning Compressed Models for Pedestrian Detection

TL;DR

This paper introduces a novel knowledge distillation approach with a hint layer and uncertainty modeling to effectively train small pedestrian detection models, achieving high accuracy with significantly fewer parameters.

Contribution

It proposes a new method combining a hint layer, uncertainty estimation, and hand-designed features to improve small model training for pedestrian detection.

Findings

Small model with 400x fewer parameters outperforms AlexNet on Caltech dataset.

Enhanced distillation method improves small model accuracy.

Incorporating uncertainty and features boosts pedestrian detection performance.

Abstract

Deep convolutional neural networks continue to advance the state-of-the-art in many domains as they grow bigger and more complex. It has been observed that many of the parameters of a large network are redundant, allowing for the possibility of learning a smaller network that mimics the outputs of the large network through a process called Knowledge Distillation. We show, however, that standard Knowledge Distillation is not effective for learning small models for the task of pedestrian detection. To improve this process, we introduce a higher-dimensional hint layer to increase information flow. We also estimate the variance in the outputs of the large network and propose a loss function to incorporate this uncertainty. Finally, we attempt to boost the complexity of the small network without increasing its size by using as input hand-designed features that have been demonstrated to be effective for pedestrian detection. We succeed in training a model that contains $400\times$ fewer parameters than the large network while outperforming AlexNet on the Caltech Pedestrian Dataset.