UniCL: A Universal Contrastive Learning Framework for Large Time Series Models

TL;DR

UniCL is a universal contrastive learning framework that pretrains large time-series models across diverse domains, using a novel trainable augmentation method based on spectral information to improve generalization and reduce bias.

Contribution

The paper introduces a scalable, domain-agnostic contrastive pretraining framework with a novel trainable spectral augmentation for time-series models.

Findings

Effective across eleven domains in experiments

High generalization demonstrated on benchmark datasets

Outperforms existing pretraining methods in robustness

Abstract

Time-series analysis plays a pivotal role across a range of critical applications, from finance to healthcare, which involves various tasks, such as forecasting and classification. To handle the inherent complexities of time-series data, such as high dimensionality and noise, traditional supervised learning methods first annotate extensive labels for time-series data in each task, which is very costly and impractical in real-world applications. In contrast, pre-trained foundation models offer a promising alternative by leveraging unlabeled data to capture general time series patterns, which can then be fine-tuned for specific tasks. However, existing approaches to pre-training such models typically suffer from high-bias and low-generality issues due to the use of predefined and rigid augmentation operations and domain-specific data training. To overcome these limitations, this paper introduces UniCL, a universal and scalable contrastive learning framework designed for pretraining time-series foundation models across cross-domain datasets. Specifically, we propose a unified and trainable time-series augmentation operation to generate pattern-preserved, diverse, and low-bias time-series data by leveraging spectral information. Besides, we introduce a scalable augmentation algorithm capable of handling datasets with varying lengths, facilitating cross-domain pretraining. Extensive experiments on two benchmark datasets across eleven domains validate the effectiveness of UniCL, demonstrating its high generalization on time-series analysis across various fields.