Multimodal Learning with Uncertainty Quantification based on Discounted Belief Fusion

TL;DR

This paper introduces a new multimodal learning approach that effectively manages uncertainty and conflicts between modalities using order-invariant evidence fusion and a conflict-based discounting mechanism, improving reliability in decision-making.

Contribution

It proposes a novel, order-invariant evidence fusion method with a conflict-aware discounting mechanism for better uncertainty management in multimodal AI.

Findings

Outperforms previous models in uncertainty-based conflict detection

Effectively distinguishes conflicting from non-conflicting samples

Demonstrates improved reliability in multimodal decision-making

Abstract

Multimodal AI models are increasingly used in fields like healthcare, finance, and autonomous driving, where information is drawn from multiple sources or modalities such as images, texts, audios, videos. However, effectively managing uncertainty - arising from noise, insufficient evidence, or conflicts between modalities - is crucial for reliable decision-making. Current uncertainty-aware machine learning methods leveraging, for example, evidence averaging, or evidence accumulation underestimate uncertainties in high-conflict scenarios. Moreover, the state-of-the-art evidence averaging strategy is not order invariant and fails to scale to multiple modalities. To address these challenges, we propose a novel multimodal learning method with order-invariant evidence fusion and introduce a conflict-based discounting mechanism that reallocates uncertain mass when unreliable modalities are detected. We provide both theoretical analysis and experimental validation, demonstrating that unlike the previous work, the proposed approach effectively distinguishes between conflicting and non-conflicting samples based on the provided uncertainty estimates, and outperforms the previous models in uncertainty-based conflict detection.