multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data

TL;DR

multivariateGPT is a transformer-based model that effectively handles sequences of mixed categorical and numeric data, enabling better modeling of complex real-world time series and physical systems.

Contribution

it introduces a unified transformer architecture for joint modeling of categorical and numeric data, extending the utility of transformers to diverse data types.

Findings

successfully models physical systems and time series

generalizes patterns in complex data

extends transformer applicability to mixed data types

Abstract

Real-world processes often generate data that are a mix of categorical and numeric values that are recorded at irregular and informative intervals. Discrete token-based approaches are limited in numeric representation capacity while methods like neural ordinary differential equations are not well suited for categorical data or informative sampling and require augmentation to handle certain classes of trajectories. Here, we present multivariateGPT, a single architecture for modeling sequences of mixed categorical (including tokenized text) and numeric data. This is accomplished with an autoregressive sequence decomposition, embedding scheme, and loss function that extend the next token prediction task to likelihood estimation of the joint distribution of next token class and value. We demonstrate how this approach can efficiently learn to generalize patterns in simple physical systems and model complex time series including electrocardiograms and multivariate electronic health record data. This work extends the utility of transformer based models to additional classes of data.