Quantifying Multimodal Imbalance: A GMM-Guided Adaptive Loss for Audio-Visual Learning

TL;DR

This paper introduces a GMM-guided adaptive loss framework for audio-visual learning that dynamically addresses modality imbalance at the sample level, improving performance and data quality.

Contribution

It proposes a novel GMM-based method to diagnose and mitigate modality imbalance dynamically, outperforming existing static approaches in multimodal learning.

Findings

Significantly outperforms state-of-the-art baselines on multiple datasets.

Effectively identifies and filters noisy samples to enhance learning.

Demonstrates improved fusion and alignment through adaptive loss.

Abstract

Multimodal learning integrates diverse modalities but suffers from modality imbalance, where dominant modalities suppress weaker ones due to inconsistent convergence rates. Existing methods predominantly rely on static modulation or heuristics, overlooking sample-level distributional variations in prediction bias. Specifically, they fail to distinguish outlier samples where the modality gap is exacerbated by low data quality. We propose a framework to quantitatively diagnose and dynamically mitigate this imbalance at the sample level. We introduce the Modality Gap metric to quantify prediction discrepancies. Analysis reveals that this gap follows a bimodal distribution, indicating the coexistence of balanced and imbalanced sample subgroups. We employ a Gaussian Mixture Model (GMM) to explicitly model this distribution, leveraging Bayesian posterior probabilities for soft subgroup separation. Our two-stage framework comprises a Warm-up stage and an Adaptive Training stage. In the latter, a GMM-guided Adaptive Loss dynamically reallocates optimization priorities: it imposes stronger alignment penalties on imbalanced samples to rectify bias, while prioritizing fusion for balanced samples to maximize complementary information. Experiments on CREMA-D, AVE, and KineticSound demonstrate that our method significantly outperforms SOTA baselines. Furthermore, we show that fine-tuning on a GMM-filtered balanced subset serves as an effective data purification strategy, yielding substantial gains by eliminating extreme noisy samples even without the adaptive loss.