Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis

TL;DR

This paper introduces MADDi, a novel multimodal deep learning framework utilizing cross-modal attention to improve Alzheimer's Disease diagnosis accuracy by integrating imaging, genetic, and clinical data, outperforming previous models.

Contribution

The study presents the first application of cross-modal attention in multimodal AD diagnosis, enhancing classification accuracy and demonstrating the importance of structured clinical data.

Findings

MADDi achieves 96.88% accuracy in classifying MCI, AD, and controls.

Cross-modal attention with self-attention outperforms other attention schemes.

Structured clinical data significantly improves model performance.

Abstract

Alzheimer's Disease (AD) is the most common neurodegenerative disorder with one of the most complex pathogeneses, making effective and clinically actionable decision support difficult. The objective of this study was to develop a novel multimodal deep learning framework to aid medical professionals in AD diagnosis. We present a Multimodal Alzheimer's Disease Diagnosis framework (MADDi) to accurately detect the presence of AD and mild cognitive impairment (MCI) from imaging, genetic, and clinical data. MADDi is novel in that we use cross-modal attention, which captures interactions between modalities - a method not previously explored in this domain. We perform multi-class classification, a challenging task considering the strong similarities between MCI and AD. We compare with previous state-of-the-art models, evaluate the importance of attention, and examine the contribution of each modality to the model's performance. MADDi classifies MCI, AD, and controls with 96.88% accuracy on a held-out test set. When examining the contribution of different attention schemes, we found that the combination of cross-modal attention with self-attention performed the best, and no attention layers in the model performed the worst, with a 7.9% difference in F1-Scores. Our experiments underlined the importance of structured clinical data to help machine learning models contextualize and interpret the remaining modalities. Extensive ablation studies showed that any multimodal mixture of input features without access to structured clinical information suffered marked performance losses. This study demonstrates the merit of combining multiple input modalities via cross-modal attention to deliver highly accurate AD diagnostic decision support.