Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillation

TL;DR

This paper investigates how domain gaps between modalities affect cross-modal knowledge distillation, revealing that smaller distribution differences in non-target classes improve performance, supported by theoretical bounds and extensive experiments.

Contribution

It introduces the Non-Target Divergence Hypothesis (NTDH), providing a theoretical framework and empirical validation for understanding domain gaps in cross-modal knowledge distillation.

Findings

Domain gaps cause distribution differences in non-target classes.

Smaller non-target class differences lead to better distillation performance.

Theoretical bounds validate the impact of domain gaps on error.

Abstract

Compared to single-modal knowledge distillation, cross-modal knowledge distillation faces more severe challenges due to domain gaps between modalities. Although various methods have proposed various solutions to overcome these challenges, there is still limited research on how domain gaps affect cross-modal knowledge distillation. This paper provides an in-depth analysis and evaluation of this issue. We first introduce the Non-Target Divergence Hypothesis (NTDH) to reveal the impact of domain gaps on cross-modal knowledge distillation. Our key finding is that domain gaps between modalities lead to distribution differences in non-target classes, and the smaller these differences, the better the performance of cross-modal knowledge distillation. Subsequently, based on Vapnik-Chervonenkis (VC) theory, we derive the upper and lower bounds of the approximation error for cross-modal knowledge distillation, thereby theoretically validating the NTDH. Finally, experiments on five cross-modal datasets further confirm the validity, generalisability, and applicability of the NTDH.