CanKD: Cross-Attention-based Non-local operation for Feature-based Knowledge Distillation

TL;DR

CanKD introduces a cross-attention-based non-local approach for feature-based knowledge distillation, enabling more comprehensive pixel-wise knowledge transfer and outperforming existing methods in vision tasks.

Contribution

The paper presents a novel cross-attention mechanism for feature distillation that captures pixel relationships more effectively than traditional self-attention methods.

Findings

Outperforms state-of-the-art distillation methods in object detection.

Improves feature representation in image segmentation.

Requires only an additional loss function for implementation.

Abstract

We propose Cross-Attention-based Non-local Knowledge Distillation (CanKD), a novel feature-based knowledge distillation framework that leverages cross-attention mechanisms to enhance the knowledge transfer process. Unlike traditional self-attention-based distillation methods that align teacher and student feature maps independently, CanKD enables each pixel in the student feature map to dynamically consider all pixels in the teacher feature map. This non-local knowledge transfer more thoroughly captures pixel-wise relationships, improving feature representation learning. Our method introduces only an additional loss function to achieve superior performance compared with existing attention-guided distillation methods. Extensive experiments on object detection and image segmentation tasks demonstrate that CanKD outperforms state-of-the-art feature and hybrid distillation methods. These experimental results highlight CanKD's potential as a new paradigm for attention-guided distillation in computer vision tasks. Code is available at https://github.com/tori-hotaru/CanKD