Quantifying Epistemic Uncertainty in Multimodal Long-Tailed Classification: A Belief Entropy-Based Evidential Fusion Framework

TL;DR

This paper introduces a new framework to handle uncertainty and improve fairness in multimodal classification tasks with imbalanced data.

Contribution

The novel framework, UMuLT, combines evidential reasoning with deep learning to address uncertainty and class imbalance in multimodal settings.

Findings

UMuLT improves performance on tail classes in long-tailed multimodal classification tasks.

The framework outperforms existing methods in overall metrics and calibration.

Statistical significance tests validate the effectiveness of the proposed approach.

Abstract

Deep multimodal learning has excelled in tasks involving vision, language, and audio modalities. Nevertheless, their performance on tail classes exhibits significant degradation under the long-tailed distributions common in real-world data, meanwhile related fusion schemes often provide only limited treatment of modality-specific uncertainty and rarely incorporate explicit mechanisms for class-level fairness. To address these information discrepancies, we present a framework that integrates evidential reasoning with deep learning–Uncertainty-Quantified Multimodal Learning for Long-Tailed Classification (UMuLT). The framework includes: (i) an uncertainty-gated evidential fusion module that adaptively down-weights unreliable modalities; (ii) an exponential moving average (EMA) fairness regularizer that dynamically amplifies tail-class gradients; and (iii) a cross-modal consistency regularizer optimized in two stages: tail specialization with lightweight adapters on tail-class data to obtain a balanced initialization, followed by end-to-end fine-tuning. The effectiveness and practicality of our method are verified on three long-tailed benchmarks for multimodal classification. Experiments show consistent gains over strong baselines in overall metrics, calibration, and tail subset performance. Statistical significance tests confirm the superiority of the proposed framework.

Figures