MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding

TL;DR

This paper introduces a saliency-aware multimodal knowledge distillation framework that improves model performance by focusing on modality-specific information and saliency-based weighting in vision-language tasks.

Contribution

It proposes a novel modality-specific distillation method with saliency-based weighting, enhancing knowledge transfer in multimodal models.

Findings

MSD outperforms traditional KD on four multimodal datasets.

Saliency-based weighting improves the effectiveness of knowledge distillation.

Modality-specific analysis reveals the importance of different modalities in KD.

Abstract

To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. However, KD on multimodal datasets such as vision-language tasks is relatively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework, modality-specific distillation (MSD), to transfer knowledge from a teacher on multimodal tasks by learning the teacher's behavior within each modality. The idea aims at mimicking a teacher's modality-specific predictions by introducing auxiliary loss terms for each modality. Furthermore, because each modality has different saliency for predictions, we define saliency scores for each modality and investigate saliency-based weighting schemes for the auxiliary losses. We further study a weight learning approach to learn the optimal weights on these loss terms. In our empirical analysis, we examine the saliency of each modality in KD, demonstrate the effectiveness of the weighting scheme in MSD, and show that it achieves better performance than KD on four multimodal datasets.