Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability

TL;DR

This paper proposes a cross-attention multimodal fusion model that combines mammography images and clinical data to improve breast cancer diagnosis, achieving high accuracy and interpretability on public datasets.

Contribution

It introduces a novel multimodal deep learning approach using cross-attention for integrating imaging and clinical data in breast cancer diagnosis.

Findings

Achieved an AUC-ROC of 0.98 on public datasets

Improved diagnostic accuracy with multimodal data

Enhanced model interpretability through explainability methods

Abstract

A precise assessment of the risk of breast lesions can greatly lower it and assist physicians in choosing the best course of action. To categorise breast lesions, the majority of current computer-aided systems only use characteristics from mammograms. Although this method is practical, it does not completely utilise clinical reports' valuable information to attain the best results. When compared to utilising mammography alone, will clinical features greatly enhance the categorisation of breast lesions? How may clinical features and mammograms be combined most effectively? In what ways may explainable AI approaches improve the interpretability and reliability of models used to diagnose breast cancer? To answer these basic problems, a comprehensive investigation is desperately needed. In order to integrate mammography and categorical clinical characteristics, this study examines a number of multimodal deep networks grounded on feature concatenation, co-attention, and cross-attention. The model achieved an AUC-ROC of 0.98, accuracy of 0.96, F1-score of 0.94, precision of 0.92, and recall of 0.95 when tested on publicly accessible datasets (TCGA and CBIS-DDSM).