Towards modeling evolving longitudinal health trajectories with a transformer-based deep learning model

TL;DR

This paper presents a transformer-based deep learning approach for modeling and analyzing evolving health trajectories over time using longitudinal health data, enabling continuous predictions and early detection of disease onset.

Contribution

It introduces a novel training method with causal attention masks for transformers to model health trajectories continuously over time.

Findings

Model performs comparably to existing models in disease prediction.

Enables continuous health trajectory analysis at every time point.

Potential for early intervention and ongoing health monitoring.

Abstract

Health registers contain rich information about individuals' health histories. Here our interest lies in understanding how individuals' health trajectories evolve in a nationwide longitudinal dataset with coded features, such as clinical codes, procedures, and drug purchases. We introduce a straightforward approach for training a Transformer-based deep learning model in a way that lets us analyze how individuals' trajectories change over time. This is achieved by modifying the training objective and by applying a causal attention mask. We focus here on a general task of predicting the onset of a range of common diseases in a given future forecast interval. However, instead of providing a single prediction about diagnoses that could occur in this forecast interval, our approach enable the model to provide continuous predictions at every time point up until, and conditioned on, the time of the forecast period. We find that this model performs comparably to other models, including a bi-directional transformer model, in terms of basic prediction performance while at the same time offering promising trajectory modeling properties. We explore a couple of ways to use this model for analyzing health trajectories and aiding in early detection of events that forecast possible later disease onsets. We hypothesize that this method may be helpful in continuous monitoring of peoples' health trajectories and enabling interventions in ongoing health trajectories, as well as being useful in retrospective analyses.