Multimodal Fusion Learning with Dual Attention for Medical Imaging

TL;DR

This paper introduces DRIFA-Net, a multimodal fusion learning framework with dual attention modules that enhances disease classification across various medical imaging modalities by improving representation learning and generalization.

Contribution

It proposes a novel dual attention mechanism, DRIFA, that can be integrated into any deep neural network to improve multimodal fusion and generalization in medical imaging tasks.

Findings

Outperforms state-of-the-art methods on five datasets

Enhances modality-specific and shared representations

Improves generalization across multiple diagnosis tasks

Abstract

Multimodal fusion learning has shown significant promise in classifying various diseases such as skin cancer and brain tumors. However, existing methods face three key limitations. First, they often lack generalizability to other diagnosis tasks due to their focus on a particular disease. Second, they do not fully leverage multiple health records from diverse modalities to learn robust complementary information. And finally, they typically rely on a single attention mechanism, missing the benefits of multiple attention strategies within and across various modalities. To address these issues, this paper proposes a dual robust information fusion attention mechanism (DRIFA) that leverages two attention modules, i.e. multi-branch fusion attention module and the multimodal information fusion attention module. DRIFA can be integrated with any deep neural network, forming a multimodal fusion learning framework denoted as DRIFA-Net. We show that the multi-branch fusion attention of DRIFA learns enhanced representations for each modality, such as dermoscopy, pap smear, MRI, and CT-scan, whereas multimodal information fusion attention module learns more refined multimodal shared representations, improving the network's generalization across multiple tasks and enhancing overall performance. Additionally, to estimate the uncertainty of DRIFA-Net predictions, we have employed an ensemble Monte Carlo dropout strategy. Extensive experiments on five publicly available datasets with diverse modalities demonstrate that our approach consistently outperforms state-of-the-art methods. The code is available at https://github.com/misti1203/DRIFA-Net.