CNN-LSTM Based Multimodal MRI and Clinical Data Fusion for Predicting Functional Outcome in Stroke Patients

TL;DR

This paper introduces a multimodal CNN-LSTM ensemble model that integrates MRI images and clinical data to predict stroke patient outcomes, demonstrating improved accuracy over baseline methods.

Contribution

The novel model effectively combines heterogeneous MRI and clinical data using a deep learning architecture for improved stroke outcome prediction.

Findings

Achieved highest AUC of 0.77 with NIHSS data.

Outperformed baseline models in predictive accuracy.

Automatically encodes spatio-temporal MRI features.

Abstract

Clinical outcome prediction plays an important role in stroke patient management. From a machine learning point-of-view, one of the main challenges is dealing with heterogeneous data at patient admission, i.e. the image data which are multidimensional and the clinical data which are scalars. In this paper, a multimodal convolutional neural network - long short-term memory (CNN-LSTM) based ensemble model is proposed. For each MR image module, a dedicated network provides preliminary prediction of the clinical outcome using the modified Rankin scale (mRS). The final mRS score is obtained by merging the preliminary probabilities of each module dedicated to a specific type of MR image weighted by the clinical metadata, here age or the National Institutes of Health Stroke Scale (NIHSS). The experimental results demonstrate that the proposed model surpasses the baselines and offers an original way to automatically encode the spatio-temporal context of MR images in a deep learning architecture. The highest AUC (0.77) was achieved for the proposed model with NIHSS.