Asymmetric Cross-Modal Knowledge Distillation: Bridging Modalities with Weak Semantic Consistency

TL;DR

This paper introduces Asymmetric Cross-modal Knowledge Distillation (ACKD), a flexible approach for bridging modalities with weak semantic overlap, and proposes SemBridge to improve knowledge transfer efficiency, validated on a new remote sensing dataset.

Contribution

The paper presents ACKD for weak semantic consistency scenarios and introduces SemBridge, combining self-supervised learning and optimal transport for effective cross-modal knowledge transfer.

Findings

Achieves state-of-the-art results on remote sensing datasets.

Demonstrates effectiveness across multiple model architectures.

Validates the approach with comprehensive experiments.

Abstract

Cross-modal Knowledge Distillation has demonstrated promising performance on paired modalities with strong semantic connections, referred to as Symmetric Cross-modal Knowledge Distillation (SCKD). However, implementing SCKD becomes exceedingly constrained in real-world scenarios due to the limited availability of paired modalities. To this end, we investigate a general and effective knowledge learning concept under weak semantic consistency, dubbed Asymmetric Cross-modal Knowledge Distillation (ACKD), aiming to bridge modalities with limited semantic overlap. Nevertheless, the shift from strong to weak semantic consistency improves flexibility but exacerbates challenges in knowledge transmission costs, which we rigorously verified based on optimal transport theory. To mitigate the issue, we further propose a framework, namely SemBridge, integrating a Student-Friendly Matching module and a Semantic-aware Knowledge Alignment module. The former leverages self-supervised learning to acquire semantic-based knowledge and provide personalized instruction for each student sample by dynamically selecting the relevant teacher samples. The latter seeks the optimal transport path by employing Lagrangian optimization. To facilitate the research, we curate a benchmark dataset derived from two modalities, namely Multi-Spectral (MS) and asymmetric RGB images, tailored for remote sensing scene classification. Comprehensive experiments exhibit that our framework achieves state-of-the-art performance compared with 7 existing approaches on 6 different model architectures across various datasets.