NuTime: Numerically Multi-Scaled Embedding for Large-Scale Time-Series Pretraining

TL;DR

NuTime introduces a numerically multi-scaled embedding approach for large-scale time-series pretraining, enabling effective semantic representation learning on datasets with millions of sequences.

Contribution

It presents a novel embedding module tailored for numerical properties of time-series data, scaling pretraining to large datasets and improving transfer performance across tasks.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Significantly outperforms previous pretraining methods.

Effective on both univariate and multivariate tasks.

Abstract

Recent research on time-series self-supervised models shows great promise in learning semantic representations. However, it has been limited to small-scale datasets, e.g., thousands of temporal sequences. In this work, we make key technical contributions that are tailored to the numerical properties of time-series data and allow the model to scale to large datasets, e.g., millions of temporal sequences. We adopt the Transformer architecture by first partitioning the input into non-overlapping windows. Each window is then characterized by its normalized shape and two scalar values denoting the mean and standard deviation within each window. To embed scalar values that may possess arbitrary numerical amplitudes in a high-dimensional space, we propose a numerically multi-scaled embedding module enumerating all possible numerical scales for the scalars. The model undergoes pretraining with a simple contrastive objective on a large-scale dataset over a million sequences collected by merging existing public data. We study its transfer performance on a number of univariate and multivariate classification tasks, few shot learning, unsupervised clustering and anomaly detection benchmarks. Our method exhibits remarkable improvement against previous pretraining approaches and establishes the new state of the art, even compared with domain-specific non-learning-based methods. Code is available at: \url{https://github.com/chenguolin/NuTime}.