Timelygpt: extrapolatable transformer pre-training for long-term time-series forecasting in healthcare

TL;DR

TimelyGPT is a new transformer model designed to accurately forecast long-term health trends from time-series data in electronic health records.

Contribution

Introduces TimelyGPT with extrapolatable position embeddings and modules for global–local temporal dependencies in healthcare time-series forecasting.

Findings

TimelyGPT accurately extrapolates body temperature trends over 6000 timesteps from a 2000-timestep prompt.

Achieves high recall scores in predicting future diagnoses from early irregular clinical records.

Demonstrates strong performance in modeling both continuous biosignals and irregularly-sampled health data.

Abstract

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success in Natural Language Processing and Computer Vision domains. However, the development of PTMs on healthcare time-series data is lagging behind. This underscores the limitations of the existing transformer-based architectures, particularly their scalability to handle large-scale time series and ability to capture long-term temporal dependencies.

In this study, we present Timely Generative Pre-trained Transformer (TimelyGPT). TimelyGPT employs an extrapolatable position (xPos) embedding to encode trend and periodic patterns into time-series representations. It also integrates recurrent attention and temporal convolution modules to effectively capture global–local temporal dependencies.

Our experiments show that TimelyGPT excels in modeling continuously monitored biosignals and irregularly-sampled time series data commonly observed in longitudinal electronic health records (EHRs). In forecasting continuous biosignals, TimelyGPT achieves accurate extrapolation up to 6000 timesteps of body temperature during the sleep stage transition given a short look-up window (i.e., prompt) containing only 2000 timesteps. For irregularly-sampled time series, TimelyGPT with a proposed time-specific inference demonstrates high top recall scores in predicting future diagnoses using early diagnostic records, effectively handling irregular intervals between clinical records. We further demonstrate that TimelyGPT achieves strong discriminative performance on both continuous and irregularly-sampled time series.

Together, we envision TimelyGPT to be useful in various health domains, including long-term patient health state forecasting, patient risk trajectory prediction, and disease classification. Its code is available at Github.

Figures