Toward a Foundation Model for Time Series Data

TL;DR

This paper develops a multi-domain time series foundation model using self-supervised pre-training, demonstrating improved classification performance and faster convergence, especially with Transformer architectures.

Contribution

It introduces a novel multi-domain pre-training approach for time series data and evaluates its effectiveness across different neural network architectures.

Findings

Pre-training enhances downstream classification accuracy.

Pre-training accelerates convergence during fine-tuning.

The proposed method with Transformers outperforms other approaches.

Abstract

A foundation model is a machine learning model trained on a large and diverse set of data, typically using self-supervised learning-based pre-training techniques, that can be adapted to various downstream tasks. However, current research on time series pre-training has mostly focused on models pre-trained solely on data from a single domain, resulting in a lack of knowledge about other types of time series. However, current research on time series pre-training has predominantly focused on models trained exclusively on data from a single domain. As a result, these models possess domain-specific knowledge that may not be easily transferable to time series from other domains. In this paper, we aim to develop an effective time series foundation model by leveraging unlabeled samples from multiple domains. To achieve this, we repurposed the publicly available UCR Archive and evaluated four existing self-supervised learning-based pre-training methods, along with a novel method, on the datasets. We tested these methods using four popular neural network architectures for time series to understand how the pre-training methods interact with different network designs. Our experimental results show that pre-training improves downstream classification tasks by enhancing the convergence of the fine-tuning process. Furthermore, we found that the proposed pre-training method, when combined with the Transformer model, outperforms the alternatives.