MILM: Large Language Models for Multimodal Irregular Time Series with Informative Sampling

TL;DR

This paper introduces MILM, a two-stage large language model fine-tuning approach for multimodal irregular time series in healthcare, effectively leveraging sampling patterns and textual data for improved predictive performance.

Contribution

The paper proposes a novel two-stage fine-tuning strategy for LLMs on MITS data, emphasizing the importance of sampling patterns and handling value-pending scenarios.

Findings

MILM-2S achieves top performance on multiple EHR datasets.

Sampling patterns contain significant predictive information.

MILM-2S outperforms MILM-Direct in value-pending evaluation.

Abstract

Multimodal irregular time series (MITS) consist of asynchronous and irregularly sampled observations from heterogeneous numerical and textual channels. In healthcare, for example, patients' electronic health records (EHR) include irregular lab measurements and clinical notes. The irregular timing and channel patterns of observations carry predictive signal alongside the numerical values and textual content. LLMs are natural candidates for processing such heterogeneous data, given their extensive pretrained knowledge spanning textual and numerical domains. We introduce MILM (Multimodal Irregular time series Language Model), which represents MITS as time-ordered triplets in Extensible Markup Language (XML) format and fine-tunes an LLM through a two-stage strategy for MITS classification. The first stage trains on value-redacted MITS to predict from sampling patterns alone, and the second stage trains on full MITS to jointly model sampling patterns and observed values. Our two-stage model (MILM-2S) and its single-stage counterpart (MILM-Direct) achieve the best and second-best average performance on multiple EHR datasets. Further value redaction evaluations confirm that sampling patterns carry predictive signal and that MILM-2S learns to exploit them. In the value pending evaluation we introduce, where some values are unavailable at prediction time, MILM-2S outperforms MILM-Direct by a larger margin compared to standard evaluation. For MILM-2S, preserving the time and channel of value-pending observations as additional sampling information further improves in-hospital mortality prediction.