Multimodal Deep Learning for Stroke Prediction and Detection using Retinal Imaging and Clinical Data

TL;DR

This study develops a multimodal deep learning model combining retinal images and clinical data to improve stroke detection and risk prediction, demonstrating significant performance gains over existing methods.

Contribution

It introduces a novel multimodal neural network leveraging retinal scans and clinical data, pretrained with self-supervised learning, for stroke prediction.

Findings

Achieved 5% AUROC improvement over image-only baseline.

Achieved 8% AUROC improvement over state-of-the-art foundation model.

Established retinal imaging's potential in stroke risk assessment.

Abstract

Stroke is a major public health problem, affecting millions worldwide. Deep learning has recently demonstrated promise for enhancing the diagnosis and risk prediction of stroke. However, existing methods rely on costly medical imaging modalities, such as computed tomography. Recent studies suggest that retinal imaging could offer a cost-effective alternative for cerebrovascular health assessment due to the shared clinical pathways between the retina and the brain. Hence, this study explores the impact of leveraging retinal images and clinical data for stroke detection and risk prediction. We propose a multimodal deep neural network that processes Optical Coherence Tomography (OCT) and infrared reflectance retinal scans, combined with clinical data, such as demographics, vital signs, and diagnosis codes. We pretrained our model using a self-supervised learning framework using a real-world dataset consisting of $37$ k scans, and then fine-tuned and evaluated the model using a smaller labeled subset. Our empirical findings establish the predictive ability of the considered modalities in detecting lasting effects in the retina associated with acute stroke and forecasting future risk within a specific time horizon. The experimental results demonstrate the effectiveness of our proposed framework by achieving $5$\% AUROC improvement as compared to the unimodal image-only baseline, and $8$\% improvement compared to an existing state-of-the-art foundation model. In conclusion, our study highlights the potential of retinal imaging in identifying high-risk patients and improving long-term outcomes.