Adaptive Redundancy Regulation for Balanced Multimodal Information Refinement

TL;DR

This paper introduces RedReg, an adaptive method for balancing multimodal learning by regulating redundancy and modality dominance, leading to improved performance and representation quality.

Contribution

RedReg employs a redundancy monitor and co-information gating to dynamically balance modalities, addressing long-term dominance and redundancy issues in multimodal training.

Findings

RedReg outperforms existing methods in most scenarios.

Ablation studies confirm the effectiveness of each component.

The method adapts to modality reliance, preserving modality-specific information.

Abstract

Multimodal learning aims to improve performance by leveraging data from multiple sources. During joint multimodal training, due to modality bias, the advantaged modality often dominates backpropagation, leading to imbalanced optimization. Existing methods still face two problems: First, the long-term dominance of the dominant modality weakens representation-output coupling in the late stages of training, resulting in the accumulation of redundant information. Second, previous methods often directly and uniformly adjust the gradients of the advantaged modality, ignoring the semantics and directionality between modalities. To address these limitations, we propose Adaptive Redundancy Regulation for Balanced Multimodal Information Refinement (RedReg), which is inspired by information bottleneck principle. Specifically, we construct a redundancy phase monitor that uses a joint criterion of effective gain growth rate and redundancy to trigger intervention only when redundancy is high. Furthermore, we design a co-information gating mechanism to estimate the contribution of the current dominant modality based on cross-modal semantics. When the task primarily relies on a single modality, the suppression term is automatically disabled to preserve modality-specific information. Finally, we project the gradient of the dominant modality onto the orthogonal complement of the joint multimodal gradient subspace and suppress the gradient according to redundancy. Experiments show that our method demonstrates superiority among current major methods in most scenarios. Ablation experiments verify the effectiveness of our method. The code is available at https://github.com/xia-zhe/RedReg.git