Building Lightweight Semantic Segmentation Models for Aerial Images Using Dual Relation Distillation

TL;DR

This paper introduces a dual relation distillation method to improve lightweight semantic segmentation models for aerial images by transferring spatial and channel relations from larger models, enhancing accuracy without extra computation.

Contribution

The paper proposes a novel dual relation distillation technique that transfers spatial and channel correlations from teacher to student models for better segmentation performance.

Findings

Significant accuracy improvements on Vaihingen, Potsdam, and Cityscapes datasets.

Effective knowledge transfer without additional computational cost.

Enhanced feature correlation modeling leads to better student model performance.

Abstract

Recently, there have been significant improvements in the accuracy of CNN models for semantic segmentation. However, these models are often heavy and suffer from low inference speed, which limits their practical application. To address this issue, knowledge distillation has emerged as a promising approach to achieve a good trade-off between segmentation accuracy and efficiency. In this paper, we propose a novel dual relation distillation (DRD) technique that transfers both spatial and channel relations in feature maps from a cumbersome model (teacher) to a compact model (student). Specifically, we compute spatial and channel relation maps separately for the teacher and student models, and then align corresponding relation maps by minimizing their distance. Since the teacher model usually learns more information and collects richer spatial and channel correlations than the student model, transferring these correlations from the teacher to the student can help the student mimic the teacher better in terms of feature distribution, thus improving the segmentation accuracy of the student model. We conduct comprehensive experiments on three segmentation datasets, including two widely adopted benchmarks in the remote sensing field (Vaihingen and Potsdam datasets) and one popular benchmark in general scene (Cityscapes dataset). The experimental results demonstrate that our novel distillation framework can significantly boost the performance of the student network without incurring extra computational overhead.