What are you looking at? Modality contribution in multimodal medical deep learning

TL;DR

This paper introduces a method to measure how much each data type contributes to a multimodal deep learning model's performance in medical applications.

Contribution

A model-agnostic method for quantifying modality contributions in multimodal deep learning is proposed.

Findings

Some models show preference for specific modalities, leading to unimodal behavior.

Certain datasets are inherently imbalanced across modalities.

The method provides detailed quantitative and visual insights into modality importance.

Abstract

High dimensional, multimodal data can nowadays be analyzed by huge deep neural networks with little effort. Several fusion methods for bringing together different modalities have been developed. Given the prevalence of high-dimensional, multimodal patient data in medicine, the development of multimodal models marks a significant advancement. However, how these models process information from individual sources in detail is still underexplored.

To this end, we implemented an occlusion-based modality contribution method that is both model- and performance agnostic. This method quantitatively measures the importance of each modality in the dataset for the model to fulfill its task. We applied our method to three different multimodal medical problems for experimental purposes.

Herein we found that some networks have modality preferences that tend to unimodal collapses, while some datasets are imbalanced from the ground up. Moreover, we provide fine-grained quantitative and visual attribute importance for each modality.

Our metric offers valuable insights that can support the advancement of multimodal model development and dataset creation. By introducing this method, we contribute to the growing field of interpretability in deep learning for multimodal research. This approach helps to facilitate the integration of multimodal AI into clinical practice. Our code is publicly available at https://github.com/ChristianGappGit/MC_MMD

Figures