Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection

TL;DR

This paper introduces a novel knowledge distillation method for dense object detection that addresses protocol inconsistencies between classification and localization tasks, leading to improved student model performance.

Contribution

The authors propose a cross-task consistent distillation approach with binary-classification maps and an IoU-based localization loss, enhancing dense object detection distillation.

Findings

Outperforms existing distillation methods in dense object detection

Effective in handling class imbalance in dense detectors

Simple yet superior to prior approaches

Abstract

Knowledge distillation (KD) has shown potential for learning compact models in dense object detection. However, the commonly used softmax-based distillation ignores the absolute classification scores for individual categories. Thus, the optimum of the distillation loss does not necessarily lead to the optimal student classification scores for dense object detectors. This cross-task protocol inconsistency is critical, especially for dense object detectors, since the foreground categories are extremely imbalanced. To address the issue of protocol differences between distillation and classification, we propose a novel distillation method with cross-task consistent protocols, tailored for the dense object detection. For classification distillation, we address the cross-task protocol inconsistency problem by formulating the classification logit maps in both teacher and student models as multiple binary-classification maps and applying a binary-classification distillation loss to each map. For localization distillation, we design an IoU-based Localization Distillation Loss that is free from specific network structures and can be compared with existing localization distillation losses. Our proposed method is simple but effective, and experimental results demonstrate its superiority over existing methods. Code is available at https://github.com/TinyTigerPan/BCKD.