Small Lesions-aware Bidirectional Multimodal Multiscale Fusion Network for Lung Disease Classification

TL;DR

This paper introduces MMCAF-Net, a novel deep learning model that effectively fuses multimodal 3D medical imaging and health record data for lung disease classification, especially improving detection of small lesions.

Contribution

The paper proposes a new multimodal fusion network with multi-scale cross-attention and lesion-specific feature extraction for improved lung disease diagnosis.

Findings

Significant accuracy improvement over existing methods

Effective handling of dimensionality differences in multimodal data

Enhanced detection of small lesions in lung images

Abstract

The diagnosis of medical diseases faces challenges such as the misdiagnosis of small lesions. Deep learning, particularly multimodal approaches, has shown great potential in the field of medical disease diagnosis. However, the differences in dimensionality between medical imaging and electronic health record data present challenges for effective alignment and fusion. To address these issues, we propose the Multimodal Multiscale Cross-Attention Fusion Network (MMCAF-Net). This model employs a feature pyramid structure combined with an efficient 3D multi-scale convolutional attention module to extract lesion-specific features from 3D medical images. To further enhance multimodal data integration, MMCAF-Net incorporates a multi-scale cross-attention module, which resolves dimensional inconsistencies, enabling more effective feature fusion. We evaluated MMCAF-Net on the Lung-PET-CT-Dx dataset, and the results showed a significant improvement in diagnostic accuracy, surpassing current state-of-the-art methods. The code is available at https://github.com/yjx1234/MMCAF-Net