Multimodal Breast Lesion Classification Using Cross-Attention Deep Networks

TL;DR

This paper explores multimodal deep learning architectures that combine mammogram images and clinical data to improve breast lesion classification accuracy, addressing challenges like missing data and optimal feature integration.

Contribution

It introduces and systematically compares cross-attention and co-attention based deep networks for multimodal breast lesion classification, demonstrating significant performance improvements.

Findings

Classification performance improved from 0.89 to 0.94 AUC.

Cross-attention models outperform simple concatenation methods.

Models maintain robustness even with missing clinical data.

Abstract

Accurate breast lesion risk estimation can significantly reduce unnecessary biopsies and help doctors decide optimal treatment plans. Most existing computer-aided systems rely solely on mammogram features to classify breast lesions. While this approach is convenient, it does not fully exploit useful information in clinical reports to achieve the optimal performance. Would clinical features significantly improve breast lesion classification compared to using mammograms alone? How to handle missing clinical information caused by variation in medical practice? What is the best way to combine mammograms and clinical features? There is a compelling need for a systematic study to address these fundamental questions. This paper investigates several multimodal deep networks based on feature concatenation, cross-attention, and co-attention to combine mammograms and categorical clinical variables. We show that the proposed architectures significantly increase the lesion classification performance (average area under ROC curves from 0.89 to 0.94). We also evaluate the model when clinical variables are missing.