SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities

TL;DR

This paper introduces SHAPE scores, based on Shapley values, to quantify the contribution and cooperation of individual modalities in multi-modal deep learning models, aiding better understanding and fusion strategies.

Contribution

The paper proposes a novel SHAPE scoring method to systematically evaluate the contribution and cooperation of modalities in multi-modal models, addressing a gap in quantification.

Findings

Multi-modal models often rely on dominant modalities when modalities are complementary.

Models exploit cross-modal cooperation when modalities are indispensable.

Early-stage fusion is preferable when modalities significantly cooperate.

Abstract

As deep learning advances, there is an ever-growing demand for models capable of synthesizing information from multi-modal resources to address the complex tasks raised from real-life applications. Recently, many large multi-modal datasets have been collected, on which researchers actively explore different methods of fusing multi-modal information. However, little attention has been paid to quantifying the contribution of different modalities within the proposed models. In this paper, we propose the {\bf SH}apley v{\bf A}lue-based {\bf PE}rceptual (SHAPE) scores that measure the marginal contribution of individual modalities and the degree of cooperation across modalities. Using these scores, we systematically evaluate different fusion methods on different multi-modal datasets for different tasks. Our experiments suggest that for some tasks where different modalities are complementary, the multi-modal models still tend to use the dominant modality alone and ignore the cooperation across modalities. On the other hand, models learn to exploit cross-modal cooperation when different modalities are indispensable for the task. In this case, the scores indicate it is better to fuse different modalities at relatively early stages. We hope our scores can help improve the understanding of how the present multi-modal models operate on different modalities and encourage more sophisticated methods of integrating multiple modalities.