Hands-on Guidance for Distilling Object Detectors

TL;DR

This paper introduces Hands-on Guidance Distillation, a comprehensive framework for object detector knowledge distillation that leverages feature hierarchies and novel mechanisms to improve accuracy and efficiency.

Contribution

It proposes a new detection distillation method that distills latent features across all stages and employs innovative mechanisms for more effective knowledge transfer.

Findings

Improved accuracy and speed trade-offs on VOC and COCO datasets

Robustness across different network structures

Enhanced knowledge absorption through novel distillation mechanisms

Abstract

Knowledge distillation can lead to deploy-friendly networks against the plagued computational complexity problem, but previous methods neglect the feature hierarchy in detectors. Motivated by this, we propose a general framework for detection distillation. Our method, called Hands-on Guidance Distillation, distills the latent knowledge of all stage features for imposing more comprehensive supervision, and focuses on the essence simultaneously for promoting more intense knowledge absorption. Specifically, a series of novel mechanisms are designed elaborately, including correspondence establishment for consistency, hands-on imitation loss measure and re-weighted optimization from both micro and macro perspectives. We conduct extensive evaluations with different distillation configurations over VOC and COCO datasets, which show better performance on accuracy and speed trade-offs. Meanwhile, feasibility experiments on different structural networks further prove the robustness of our HGD.