Cross-Modal Distillation For Widely Differing Modalities

TL;DR

This paper introduces a cross-modal distillation framework that enables knowledge transfer between vastly different modalities like images, text, and speech, addressing domain gaps and overfitting issues in multi-modal learning.

Contribution

It proposes two soft constrained distillation strategies and a quality-based adaptive weighting module to improve cross-modal knowledge transfer and robustness.

Findings

Effective knowledge transfer across image, text, and speech modalities.

Reduces overfitting in cross-modal distillation.

Improves performance in speaker recognition and image classification.

Abstract

Deep learning achieved great progress recently, however, it is not easy or efficient to further improve its performance by increasing the size of the model. Multi-modal learning can mitigate this challenge by introducing richer and more discriminative information as input. To solve the problem of limited access to multi-modal data at the time of use, we conduct multi-modal learning by introducing a teacher model to transfer discriminative knowledge to a student model during training. However, this knowledge transfer via distillation is not trivial because the big domain gap between the widely differing modalities can easily lead to overfitting. In this work, we introduce a cross-modal distillation framework. Specifically, we find hard constrained loss, e.g. l2 loss forcing the student being exact the same as the teacher, can easily lead to overfitting in cross-modality distillation. To address this, we propose two soft constrained knowledge distillation strategies at the feature level and classifier level respectively. In addition, we propose a quality-based adaptive weights module to weigh input samples via quantified data quality, leading to robust model training. We conducted experiments on speaker recognition and image classification tasks, and the results show that our approach is able to effectively achieve knowledge transfer between the commonly used and widely differing modalities of image, text, and speech.