SAMKD: Spatial-aware Adaptive Masking Knowledge Distillation for Object Detection

TL;DR

SAMKD introduces a spatially hierarchical and adaptive masking distillation framework that enhances local detail transfer for improved object detection accuracy, outperforming existing methods.

Contribution

The paper proposes a novel spatial-aware adaptive masking distillation method with hierarchical feature masking and region-specific guidance for better knowledge transfer in object detection.

Findings

Improves student detector mAP from 35.3% to 38.8%.

Outperforms state-of-the-art distillation methods like MGD, FreeKD, DMKD.

Demonstrates effectiveness across different teacher-student configurations.

Abstract

Most of recent attention-guided feature masking distillation methods perform knowledge transfer via global teacher attention maps without delving into fine-grained clues. Instead, performing distillation at finer granularity is conducive to uncovering local details supplementary to global knowledge transfer and reconstructing comprehensive student features. In this study, we propose a Spatial-aware Adaptive Masking Knowledge Distillation (SAMKD) framework for accurate object detection. Different from previous feature distillation methods which mainly perform single-scale feature masking, we develop spatially hierarchical feature masking distillation scheme, such that the object-aware locality is encoded during coarse-to-fine distillation process for improved feature reconstruction. In addition, our spatial-aware feature distillation strategy is combined with a masking logit distillation scheme in which region-specific feature difference between teacher and student networks is utilized to adaptively guide the distillation process. Thus, it can help the student model to better learn from the teacher counterpart with improved knowledge transfer and reduced gap. Extensive experiments for detection task demonstrate the superiority of our method. For example, when FCOS is used as teacher detector with ResNet101 backbone, our method improves the student network from 35.3\% to 38.8\% mAP, outperforming state-of-the-art distillation methods including MGD, FreeKD and DMKD.