A Dimensional Structure based Knowledge Distillation Method for Cross-Modal Learning

TL;DR

This paper introduces a novel cross-modal knowledge distillation method based on dimensional structure analysis, improving learning performance in challenging visual tasks by enforcing feature independence and distribution uniformity.

Contribution

It proposes a new DS-based CMKD approach that enhances cross-modal learning by leveraging feature correlation and distribution, and provides a real-world dataset for community use.

Findings

Improved accuracy in cross-modal tasks with large modality gaps

Effective feature disentanglement through channel-wise independence

Validated on real-world and benchmark datasets

Abstract

Due to limitations in data quality, some essential visual tasks are difficult to perform independently. Introducing previously unavailable information to transfer informative dark knowledge has been a common way to solve such hard tasks. However, research on why transferred knowledge works has not been extensively explored. To address this issue, in this paper, we discover the correlation between feature discriminability and dimensional structure (DS) by analyzing and observing features extracted from simple and hard tasks. On this basis, we express DS using deep channel-wise correlation and intermediate spatial distribution, and propose a novel cross-modal knowledge distillation (CMKD) method for better supervised cross-modal learning (CML) performance. The proposed method enforces output features to be channel-wise independent and intermediate ones to be uniformly distributed, thereby learning semantically irrelevant features from the hard task to boost its accuracy. This is especially useful in specific applications where the performance gap between dual modalities is relatively large. Furthermore, we collect a real-world CML dataset to promote community development. The dataset contains more than 10,000 paired optical and radar images and is continuously being updated. Experimental results on real-world and benchmark datasets validate the effectiveness of the proposed method.