PAITS: Pretraining and Augmentation for Irregularly-Sampled Time Series

TL;DR

PAITS is a framework that improves pretraining and augmentation techniques specifically for irregularly sampled and sparse time series data, leveraging NLP-inspired tasks and dataset-specific strategies.

Contribution

The paper introduces PAITS, a novel framework combining NLP-inspired pretraining and augmentation with a dataset-specific search to enhance irregular time series modeling.

Findings

Different datasets require different pretraining strategies.

PAITS consistently outperforms prior methods across multiple datasets.

The approach improves pretraining effectiveness for irregularly sampled data.

Abstract

Real-world time series data that commonly reflect sequential human behavior are often uniquely irregularly sampled and sparse, with highly nonuniform sampling over time and entities. Yet, commonly-used pretraining and augmentation methods for time series are not specifically designed for such scenarios. In this paper, we present PAITS (Pretraining and Augmentation for Irregularly-sampled Time Series), a framework for identifying suitable pretraining strategies for sparse and irregularly sampled time series datasets. PAITS leverages a novel combination of NLP-inspired pretraining tasks and augmentations, and a random search to identify an effective strategy for a given dataset. We demonstrate that different datasets benefit from different pretraining choices. Compared with prior methods, our approach is better able to consistently improve pretraining across multiple datasets and domains. Our code is available at \url{https://github.com/google-research/google-research/tree/master/irregular_timeseries_pretraining}.