Enhancing multimodal cooperation via sample-level modality valuation

TL;DR

This paper introduces a sample-level modality valuation metric to better understand and improve the cooperation of different modalities in multimodal learning, leading to significant performance gains.

Contribution

It proposes a novel sample-level modality valuation method to analyze and enhance multimodal cooperation at the sample level with theoretical support.

Findings

Sample-level modality discrepancy varies across samples.

Enhancing low-contributing modalities improves overall performance.

The method achieves significant improvements in multimodal cooperation.

Abstract

One primary topic of multimodal learning is to jointly incorporate heterogeneous information from different modalities. However most models often suffer from unsatisfactory multimodal cooperation which cannot jointly utilize all modalities well. Some methods are proposed to identify and enhance the worse learnt modality but they are often hard to provide the fine-grained observation of multimodal cooperation at sample-level with theoretical support. Hence it is essential to reasonably observe and improve the fine-grained cooperation between modalities especially when facing realistic scenarios where the modality discrepancy could vary across different samples. To this end we introduce a sample-level modality valuation metric to evaluate the contribution of each modality for each sample. Via modality valuation we observe that modality discrepancy indeed could be different at sample-level beyond the global contribution discrepancy at dataset-level. We further analyze this issue and improve cooperation between modalities at sample-level by enhancing the discriminative ability of low-contributing modalities in a targeted manner. Overall our methods reasonably observe the fine-grained uni-modal contribution and achieve considerable improvement. The source code and dataset are available at https://github.com/GeWu-Lab/Valuate-and-Enhance-Multimodal-Cooperation.