AIM: Adaptive Intra-Network Modulation for Balanced Multimodal Learning

TL;DR

This paper introduces AIM, a novel method for balanced multimodal learning that adaptively modulates network parameters to prevent dominance of certain modalities, leading to improved performance across benchmarks.

Contribution

AIM addresses optimization bias in multimodal learning by decoupling dominant modality parameters and adaptively modulating across network depths, a novel approach for balanced learning.

Findings

Outperforms state-of-the-art methods on multiple benchmarks.

Demonstrates strong generalizability across backbones, fusion strategies, and optimizers.

Effectively balances modality learning without hindering dominant or weak modalities.

Abstract

Multimodal learning has significantly enhanced machine learning performance but still faces numerous challenges and limitations. Imbalanced multimodal learning is one of the problems extensively studied in recent works and is typically mitigated by modulating the learning of each modality. However, we find that these methods typically hinder the dominant modality's learning to promote weaker modalities, which affects overall multimodal performance. We analyze the cause of this issue and highlight a commonly overlooked problem: optimization bias within networks. To address this, we propose Adaptive Intra-Network Modulation (AIM) to improve balanced modality learning. AIM accounts for differences in optimization state across parameters and depths within the network during modulation, achieving balanced multimodal learning without hindering either dominant or weak modalities for the first time. Specifically, AIM decouples the dominant modality's under-optimized parameters into Auxiliary Blocks and encourages reliance on these performance-degraded blocks for joint training with weaker modalities. This approach effectively prevents suppression of weaker modalities while enabling targeted optimization of under-optimized parameters to improve the dominant modality. Additionally, AIM assesses modality imbalance level across network depths and adaptively adjusts modulation strength at each depth. Experimental results demonstrate that AIM outperforms state-of-the-art imbalanced modality learning methods across multiple benchmarks and exhibits strong generalizability across different backbones, fusion strategies, and optimizers.