Multi-Modal AI for Remote Patient Monitoring in Cancer Care

TL;DR

This paper presents a multi-modal AI system for remote monitoring of cancer patients, integrating diverse data sources to predict adverse events and enable proactive care, based on a large-scale observational trial.

Contribution

It introduces a novel multi-modal AI framework tailored for real-world, asynchronous healthcare data, demonstrating its effectiveness in predicting patient risks in cancer care.

Findings

Achieved 83.9% accuracy in risk forecasting

Identified key predictive features like treatments and heart rate

Demonstrated early warning capabilities through case studies

Abstract

For patients undergoing systemic cancer therapy, the time between clinic visits is full of uncertainties and risks of unmonitored side effects. To bridge this gap in care, we developed and prospectively trialed a multi-modal AI framework for remote patient monitoring (RPM). This system integrates multi-modal data from the HALO-X platform, such as demographics, wearable sensors, daily surveys, and clinical events. Our observational trial is one of the largest of its kind and has collected over 2.1 million data points (6,080 patient-days) of monitoring from 84 patients. We developed and adapted a multi-modal AI model to handle the asynchronous and incomplete nature of real-world RPM data, forecasting a continuous risk of future adverse events. The model achieved an accuracy of 83.9% (AUROC=0.70). Notably, the model identified previous treatments, wellness check-ins, and daily maximum heart rate as key predictive features. A case study demonstrated the model's ability to provide early warnings by outputting escalating risk profiles prior to the event. This work establishes the feasibility of multi-modal AI RPM for cancer care and offers a path toward more proactive patient support.(Accepted at Europe NeurIPS 2025 Multimodal Representation Learning for Healthcare Workshop. Best Paper Poster Award.)