TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysis

TL;DR

TrajGPT is a novel Transformer model designed for irregular health time-series data, capturing continuous dynamics and enabling accurate forecasting, interpolation, and extrapolation without task-specific fine-tuning.

Contribution

It introduces the Selective Recurrent Attention mechanism and interprets TrajGPT as discretized ODEs for effective irregular time-series modeling.

Findings

Outperforms existing models in trajectory forecasting and disease prediction

Can interpolate and extrapolate disease trajectories from partial data

Effectively captures continuous health dynamics

Abstract

In many domains, such as healthcare, time-series data is often irregularly sampled with varying intervals between observations. This poses challenges for classical time-series models that require equally spaced data. To address this, we propose a novel time-series Transformer called Trajectory Generative Pre-trained Transformer (TrajGPT). TrajGPT employs a novel Selective Recurrent Attention (SRA) mechanism, which utilizes a data-dependent decay to adaptively filter out irrelevant past information based on contexts. By interpreting TrajGPT as discretized ordinary differential equations (ODEs), it effectively captures the underlying continuous dynamics and enables time-specific inference for forecasting arbitrary target timesteps. Experimental results demonstrate that TrajGPT excels in trajectory forecasting, drug usage prediction, and phenotype classification without requiring task-specific fine-tuning. By evolving the learned continuous dynamics, TrajGPT can interpolate and extrapolate disease risk trajectories from partially-observed time series. The visualization of predicted health trajectories shows that TrajGPT forecasts unseen diseases based on the history of clinically relevant phenotypes (i.e., contexts).