G$^{2}$D: Boosting Multimodal Learning with Gradient-Guided Distillation

TL;DR

G$^{2}$D introduces a gradient-guided distillation framework with dynamic modality prioritization to improve multimodal learning by balancing contributions from all modalities, especially weaker ones.

Contribution

This paper proposes G$^{2}$D, a novel knowledge distillation method with dynamic modality prioritization to address modality imbalance in multimodal learning.

Findings

Outperforms state-of-the-art methods in classification tasks.

Enhances the contribution of weak modalities during training.

Validated on multiple real-world datasets.

Abstract

Multimodal learning aims to leverage information from diverse data modalities to achieve more comprehensive performance. However, conventional multimodal models often suffer from modality imbalance, where one or a few modalities dominate model optimization, leading to suboptimal feature representation and underutilization of weak modalities. To address this challenge, we introduce Gradient-Guided Distillation (G$^{2}$D), a knowledge distillation framework that optimizes the multimodal model with a custom-built loss function that fuses both unimodal and multimodal objectives. G$^{2}$D further incorporates a dynamic sequential modality prioritization (SMP) technique in the learning process to ensure each modality leads the learning process, avoiding the pitfall of stronger modalities overshadowing weaker ones. We validate G$^{2}$D on multiple real-world datasets and show that G$^{2}$D amplifies the significance of weak modalities while training and outperforms state-of-the-art methods in classification and regression tasks. Our code is available at https://github.com/rAIson-Lab/G2D.