Attention-guided Feature Distillation for Semantic Segmentation

TL;DR

This paper introduces a simple yet effective attention-guided feature distillation method for semantic segmentation that leverages CBAM to refine feature maps, significantly improving student model performance across multiple datasets.

Contribution

It proposes a novel, straightforward knowledge distillation approach using attention mechanisms and MSE loss, outperforming complex existing methods in semantic segmentation tasks.

Findings

Achieves state-of-the-art mIoU on PascalVoc 2012, Cityscapes, COCO, and CamVid datasets.

Outperforms existing distillation methods in semantic segmentation.

Utilizes CBAM for effective feature refinement in knowledge transfer.

Abstract

Deep learning models have achieved significant results across various computer vision tasks. However, due to the large number of parameters in these models, deploying them in real-time scenarios is a critical challenge, specifically in dense prediction tasks such as semantic segmentation. Knowledge distillation has emerged as a successful technique for addressing this problem by transferring knowledge from a cumbersome model (teacher) to a lighter model (student). In contrast to existing complex methodologies commonly employed for distilling knowledge from a teacher to a student, this paper showcases the efficacy of a simple yet powerful method for utilizing refined feature maps to transfer attention. The proposed method has proven to be effective in distilling rich information, outperforming existing methods in semantic segmentation as a dense prediction task. The proposed Attention-guided Feature Distillation (AttnFD) method, employs the Convolutional Block Attention Module (CBAM), which refines feature maps by taking into account both channel-specific and spatial information content. Simply using the Mean Squared Error (MSE) loss function between the refined feature maps of the teacher and the student, AttnFD demonstrates outstanding performance in semantic segmentation, achieving state-of-the-art results in terms of improving the mean Intersection over Union (mIoU) of the student network on the PascalVoc 2012, Cityscapes, COCO, and CamVid datasets.