Rethinking Large Language Models For Irregular Time Series Classification In Critical Care

TL;DR

This paper evaluates the effectiveness of large language models for irregular ICU time series classification, highlighting the importance of encoder design and revealing current limitations in training efficiency and few-shot learning performance.

Contribution

It systematically assesses LLM components on ICU data, demonstrating that specialized encoders significantly improve performance over vanilla Transformers.

Findings

Encoder design is more critical than alignment strategy.

Specialized encoders improve AUPRC by 12.8% over vanilla Transformer.

LLM methods require 10x longer training and underperform in few-shot settings.

Abstract

Time series data from the Intensive Care Unit (ICU) provides critical information for patient monitoring. While recent advancements in applying Large Language Models (LLMs) to time series modeling (TSM) have shown great promise, their effectiveness on the irregular ICU data, characterized by particularly high rates of missing values, remains largely unexplored. This work investigates two key components underlying the success of LLMs for TSM: the time series encoder and the multimodal alignment strategy. To this end, we establish a systematic testbed to evaluate their impact across various state-of-the-art LLM-based methods on benchmark ICU datasets against strong supervised and self-supervised baselines. Results reveal that the encoder design is more critical than the alignment strategy. Encoders that explicitly model irregularity achieve substantial performance gains, yielding an average AUPRC increase of $12.8\%$ over the vanilla Transformer. While less impactful, the alignment strategy is also noteworthy, with the best-performing semantically rich, fusion-based strategy achieving a modest $2.9\%$ improvement over cross-attention. However, LLM-based methods require at least 10$\times$ longer training than the best-performing irregular supervised models, while delivering only comparable performance. They also underperform in data-scarce few-shot learning settings. These findings highlight both the promise and current limitations of LLMs for irregular ICU time series. The code is available at https://github.com/mHealthUnimelb/LLMTS.