MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis

TL;DR

MedTsLLM introduces a multimodal LLM framework that integrates time series data and textual context to perform critical physiological signal analysis tasks, improving clinical decision-making capabilities.

Contribution

This work presents a novel multimodal LLM approach tailored for medical time series analysis, effectively combining raw signals and textual information for enhanced clinical insights.

Findings

Outperforms existing models in ECG and respiratory waveform analysis

Effectively integrates multivariate time series with textual patient data

Achieves superior results in segmentation, boundary detection, and anomaly detection

Abstract

The complexity and heterogeneity of data in many real-world applications pose significant challenges for traditional machine learning and signal processing techniques. For instance, in medicine, effective analysis of diverse physiological signals is crucial for patient monitoring and clinical decision-making and yet highly challenging. We introduce MedTsLLM, a general multimodal large language model (LLM) framework that effectively integrates time series data and rich contextual information in the form of text to analyze physiological signals, performing three tasks with clinical relevance: semantic segmentation, boundary detection, and anomaly detection in time series. These critical tasks enable deeper analysis of physiological signals and can provide actionable insights for clinicians. We utilize a reprogramming layer to align embeddings of time series patches with a pretrained LLM's embedding space and make effective use of raw time series, in conjunction with textual context. Given the multivariate nature of medical datasets, we develop methods to handle multiple covariates. We additionally tailor the text prompt to include patient-specific information. Our model outperforms state-of-the-art baselines, including deep learning models, other LLMs, and clinical methods across multiple medical domains, specifically electrocardiograms and respiratory waveforms. MedTsLLM presents a promising step towards harnessing the power of LLMs for medical time series analysis that can elevate data-driven tools for clinicians and improve patient outcomes.