Asymmetric Reinforcing against Multi-modal Representation Bias

TL;DR

This paper introduces ARM, a novel asymmetric reinforcement method that dynamically balances modality contributions in multimodal learning, addressing representation bias and improving overall performance.

Contribution

The paper proposes ARM, a method that adaptively reinforces weak modalities while preserving dominant ones, effectively mitigating multimodal bias and enhancing learning outcomes.

Findings

Significant performance improvements in multimodal tasks.

Effective reduction of modality contribution imbalance.

Enhanced utilization of full multimodal information.

Abstract

The strength of multimodal learning lies in its ability to integrate information from various sources, providing rich and comprehensive insights. However, in real-world scenarios, multi-modal systems often face the challenge of dynamic modality contributions, the dominance of different modalities may change with the environments, leading to suboptimal performance in multimodal learning. Current methods mainly enhance weak modalities to balance multimodal representation bias, which inevitably optimizes from a partialmodality perspective, easily leading to performance descending for dominant modalities. To address this problem, we propose an Asymmetric Reinforcing method against Multimodal representation bias (ARM). Our ARM dynamically reinforces the weak modalities while maintaining the ability to represent dominant modalities through conditional mutual information. Moreover, we provide an in-depth analysis that optimizing certain modalities could cause information loss and prevent leveraging the full advantages of multimodal data. By exploring the dominance and narrowing the contribution gaps between modalities, we have significantly improved the performance of multimodal learning, making notable progress in mitigating imbalanced multimodal learning.