Series2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classification

TL;DR

Series2Vec introduces a self-supervised learning method for time series classification that predicts similarity in both spectral and temporal domains, outperforming existing methods and rivaling supervised approaches.

Contribution

The paper proposes Series2Vec, a novel similarity-based self-supervised learning approach for time series that does not rely on handcrafted data augmentation and uses order-invariant attention.

Findings

Outperforms current state-of-the-art self-supervised methods on nine datasets.

Performs comparably to fully supervised training with limited labels.

Fusion with other models enhances classification performance.

Abstract

We argue that time series analysis is fundamentally different in nature to either vision or natural language processing with respect to the forms of meaningful self-supervised learning tasks that can be defined. Motivated by this insight, we introduce a novel approach called \textit{Series2Vec} for self-supervised representation learning. Unlike other self-supervised methods in time series, which carry the risk of positive sample variants being less similar to the anchor sample than series in the negative set, Series2Vec is trained to predict the similarity between two series in both temporal and spectral domains through a self-supervised task. Series2Vec relies primarily on the consistency of the unsupervised similarity step, rather than the intrinsic quality of the similarity measurement, without the need for hand-crafted data augmentation. To further enforce the network to learn similar representations for similar time series, we propose a novel approach that applies order-invariant attention to each representation within the batch during training. Our evaluation of Series2Vec on nine large real-world datasets, along with the UCR/UEA archive, shows enhanced performance compared to current state-of-the-art self-supervised techniques for time series. Additionally, our extensive experiments show that Series2Vec performs comparably with fully supervised training and offers high efficiency in datasets with limited-labeled data. Finally, we show that the fusion of Series2Vec with other representation learning models leads to enhanced performance for time series classification. Code and models are open-source at \url{https://github.com/Navidfoumani/Series2Vec.}