Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation

TL;DR

This paper introduces inconsistent knowledge distillation (IKD) for object detection, leveraging data augmentation to transfer the teacher model's counter-intuitive perceptions of frequency and non-robust features, improving detection performance.

Contribution

The paper proposes a novel IKD approach that distills the teacher model's unconventional perceptions using diverse data augmentation techniques, surpassing existing KD methods.

Findings

Outperforms state-of-the-art KD baselines by up to +1.0 mAP

Effectively captures teacher's perceptions of frequency and non-robust features

Enhances various object detectors with diverse augmentation strategies

Abstract

Knowledge Distillation (KD) for object detection aims to train a compact detector by transferring knowledge from a teacher model. Since the teacher model perceives data in a way different from humans, existing KD methods only distill knowledge that is consistent with labels annotated by human expert while neglecting knowledge that is not consistent with human perception, which results in insufficient distillation and sub-optimal performance. In this paper, we propose inconsistent knowledge distillation (IKD), which aims to distill knowledge inherent in the teacher model's counter-intuitive perceptions. We start by considering the teacher model's counter-intuitive perceptions of frequency and non-robust features. Unlike previous works that exploit fine-grained features or introduce additional regularizations, we extract inconsistent knowledge by providing diverse input using data augmentation. Specifically, we propose a sample-specific data augmentation to transfer the teacher model's ability in capturing distinct frequency components and suggest an adversarial feature augmentation to extract the teacher model's perceptions of non-robust features in the data. Extensive experiments demonstrate the effectiveness of our method which outperforms state-of-the-art KD baselines on one-stage, two-stage and anchor-free object detectors (at most +1.0 mAP). Our codes will be made available at \url{https://github.com/JWLiang007/IKD.git}.