Learnable Instance Attention Filtering for Adaptive Detector Distillation

TL;DR

This paper introduces LIAF-KD, a learnable instance attention filtering framework that adaptively reweights object instances during detector knowledge distillation, leading to improved performance on KITTI and COCO datasets.

Contribution

It presents a novel learnable instance selector mechanism that dynamically evaluates instance importance, enhancing detector distillation beyond heuristic or teacher-driven methods.

Findings

Achieves a 2% performance gain on GFL ResNet-50 without extra complexity.

Demonstrates consistent improvements on KITTI and COCO datasets.

Outperforms existing state-of-the-art distillation methods.

Abstract

As deep vision models grow increasingly complex to achieve higher performance, deployment efficiency has become a critical concern. Knowledge distillation (KD) mitigates this issue by transferring knowledge from large teacher models to compact student models. While many feature-based KD methods rely on spatial filtering to guide distillation, they typically treat all object instances uniformly, ignoring instance-level variability. Moreover, existing attention filtering mechanisms are typically heuristic or teacher-driven, rather than learned with the student. To address these limitations, we propose Learnable Instance Attention Filtering for Adaptive Detector Distillation (LIAF-KD), a novel framework that introduces learnable instance selectors to dynamically evaluate and reweight instance importance during distillation. Notably, the student contributes to this process based on its evolving learning state. Experiments on the KITTI and COCO datasets demonstrate consistent improvements, with a 2% gain on a GFL ResNet-50 student without added complexity, outperforming state-of-the-art methods.