Bidirectional Generative Pre-training for Improving Healthcare Time-series Representation Learning

TL;DR

This paper introduces BiTimelyGPT, a bidirectional pre-training model for healthcare time-series data that improves representation learning and predictive performance by using alternating next and previous token prediction tasks.

Contribution

The paper presents a novel bidirectional pre-training architecture for healthcare time-series data that enhances representation quality and predictive accuracy over existing unidirectional methods.

Findings

BiTimelyGPT outperforms existing models in healthcare prediction tasks.

The model's attention maps identify key discriminative segments.

Bidirectional pre-training improves data distribution preservation.

Abstract

Learning time-series representations for discriminative tasks, such as classification and regression, has been a long-standing challenge in the healthcare domain. Current pre-training methods are limited in either unidirectional next-token prediction or randomly masked token prediction. We propose a novel architecture called Bidirectional Timely Generative Pre-trained Transformer (BiTimelyGPT), which pre-trains on biosignals and longitudinal clinical records by both next-token and previous-token prediction in alternating transformer layers. This pre-training task preserves original distribution and data shapes of the time-series. Additionally, the full-rank forward and backward attention matrices exhibit more expressive representation capabilities. Using biosignals and longitudinal clinical records, BiTimelyGPT demonstrates superior performance in predicting neurological functionality, disease diagnosis, and physiological signs. By visualizing the attention heatmap, we observe that the pre-trained BiTimelyGPT can identify discriminative segments from biosignal time-series sequences, even more so after fine-tuning on the task.