Contrast All the Time: Learning Time Series Representation from Temporal Consistency

TL;DR

CaTT introduces a novel contrastive learning method for time series that leverages the full temporal dimension and a scalable NT-pair loss, resulting in superior embeddings and faster training.

Contribution

It proposes CaTT, a contrastive learning framework that contrasts all time steps simultaneously without data augmentation, improving efficiency and embedding quality.

Findings

Produces better downstream task performance.

Faster training compared to existing methods.

Effective use of natural temporal structure.

Abstract

Representation learning for time series using contrastive learning has emerged as a critical technique for improving the performance of downstream tasks. To advance this effective approach, we introduce CaTT (\textit{Contrast All The Time}), a new approach to unsupervised contrastive learning for time series, which takes advantage of dynamics between temporally similar moments more efficiently and effectively than existing methods. CaTT departs from conventional time-series contrastive approaches that rely on data augmentations or selected views. Instead, it uses the full temporal dimension by contrasting all time steps in parallel. This is made possible by a scalable NT-pair formulation, which extends the classic N-pair loss across both batch and temporal dimensions, making the learning process end-to-end and more efficient. CaTT learns directly from the natural structure of temporal data, using repeated or adjacent time steps as implicit supervision, without the need for pair selection heuristics. We demonstrate that this approach produces superior embeddings which allow better performance in downstream tasks. Additionally, training is faster than other contrastive learning approaches, making it suitable for large-scale and real-world time series applications. The source code is publicly available at \href{https://github.com/sfi-norwai/CaTT}{https://github.com/sfi-norwai/CaTT}.