Uncertainty-Aware Vision-Language Segmentation for Medical Imaging

TL;DR

This paper presents an uncertainty-aware multimodal segmentation framework combining radiological images and clinical text, improving accuracy and reliability in medical diagnosis with efficient cross-modal fusion and uncertainty modeling.

Contribution

It introduces a novel Modality Decoding Attention Block and Spectral-Entropic Uncertainty Loss for enhanced multimodal medical image segmentation.

Findings

Achieves superior segmentation performance on multiple datasets

Demonstrates improved reliability in poor image quality conditions

Offers computational efficiency over existing state-of-the-art methods

Abstract

We introduce a novel uncertainty-aware multimodal segmentation framework that leverages both radiological images and associated clinical text for precise medical diagnosis. We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion and long-range dependency modelling. To guide learning under ambiguity, we propose the Spectral-Entropic Uncertainty (SEU) Loss, which jointly captures spatial overlap, spectral consistency, and predictive uncertainty in a unified objective. In complex clinical circumstances with poor image quality, this formulation improves model reliability. Extensive experiments on various publicly available medical datasets, QATA-COVID19, MosMed++, and Kvasir-SEG, demonstrate that our method achieves superior segmentation performance while being significantly more computationally efficient than existing State-of-the-Art (SoTA) approaches. Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks. Code: https://github.com/arya-domain/UA-VLS