Seeking Necessary and Sufficient Information from Multimodal Medical Data

TL;DR

This paper introduces a novel approach to learning multimodal medical data representations by focusing on features that are both necessary and sufficient for accurate decision-making, improving robustness and interpretability.

Contribution

We extend the Probability of Necessity and Sufficiency (PNS) framework to multimodal data by decomposing representations, enabling the learning of essential predictive features in medical applications.

Findings

Effective in synthetic datasets

Improves robustness to missing modalities

Enhances interpretability of multimodal models

Abstract

Learning multimodal representations from medical images and other data sources can provide richer information for decision-making. While various multimodal models have been developed for this, they overlook learning features that are both necessary (must be present for the outcome to occur) and sufficient (enough to determine the outcome). We argue learning such features is crucial as they can improve model performance by capturing essential predictive information, and enhance model robustness to missing modalities as each modality can provide adequate predictive signals. Such features can be learned by leveraging the Probability of Necessity and Sufficiency (PNS) as a learning objective, an approach that has proven effective in unimodal settings. However, extending PNS to multimodal scenarios remains underexplored and is non-trivial as key conditions of PNS estimation are violated. We address this by decomposing multimodal representations into modality-invariant and modality-specific components, then deriving tractable PNS objectives for each. Experiments on synthetic and real-world medical datasets demonstrate our method's effectiveness. Code will be available on GitHub.