ITGPT: Generative Pretraining on Irregular Timeseries

TL;DR

ITGPT is a novel attention-based model that effectively processes irregular, multimodal timeseries data using self-supervised and generative pretraining, achieving state-of-the-art results in healthcare and predictive maintenance tasks.

Contribution

The paper introduces ITGPT, a flexible architecture that handles irregular timeseries with SSL and GPT objectives, eliminating the need for resampling or explicit imputation.

Findings

ITGPT achieves state-of-the-art performance on healthcare and maintenance datasets.

ITGPT outperforms purely supervised models when labeled data is scarce.

The model effectively leverages unlabeled data through SSL and GPT training.

Abstract

Timeseries regression models often struggle to leverage large volumes of labeled multimodal data, particularly when the data are irregularly sampled or contain missing values. This is common in domains like healthcare and predictive maintenance, where data are collected from unreliable sources, and labeling requires expert knowledge or costly equipments. Transformer-based large language models have proven effective on structured data such as text through self-supervised learning (SSL) and generative pretraining (GPT) frameworks. However, such models lack the flexibility to efficiently process irregularly sampled multimodal timeseries data. In this paper, we introduce ITGPT, an attention-based architecture designed for handling multimodal, irregularly sampled timeseries by allowing training with both SSL losses and GPT-like objectives. We evaluate its performance on a healthcare task with the TIHM dataset, and a predictive maintenance task with the CompX dataset. Our results demonstrate that ITGPT achieves state-of-the-art performance without requiring resampling, feature fusion or explicit data imputation. Furthermore, when labels are scarce, ITGPT effectively leverages unlabeled data through SSL and GPT training, outperforming the purely supervised approach. This represents an important step towards efficiently using large and unstructured timeseries datasets for practical inference tasks.