A theoretical framework for self-supervised contrastive learning for continuous dependent data

TL;DR

This paper introduces a new theoretical framework for self-supervised contrastive learning tailored to continuous dependent data, addressing complex correlations in temporal and spatio-temporal domains.

Contribution

It develops dependency-aware loss functions and similarity measures, enabling contrastive SSL to better capture dependencies in continuous data, validated on real-world benchmarks.

Findings

Outperforms TS2Vec on UEA and UCR benchmarks by 4.17% and 2.08%.

Achieves 7% higher ROC-AUC on drought classification.

Introduces dependency-aware similarity matrices for contrastive learning.

Abstract

Self-supervised learning (SSL) has emerged as a powerful approach to learning representations, particularly in the field of computer vision. However, its application to dependent data, such as temporal and spatio-temporal domains, remains underexplored. Besides, traditional contrastive SSL methods often assume \emph{semantic independence between samples}, which does not hold for dependent data exhibiting complex correlations. We propose a novel theoretical framework for contrastive SSL tailored to \emph{continuous dependent data}, which allows the nearest samples to be semantically close to each other. In particular, we propose two possible \textit{ground truth similarity measures} between objects -- \emph{hard} and \emph{soft} closeness. Under it, we derive an analytical form for the \textit{estimated similarity matrix} that accommodates both types of closeness between samples, thereby introducing dependency-aware loss functions. We validate our approach, \emph{Dependent TS2Vec}, on temporal and spatio-temporal downstream problems. Given the dependency patterns presented in the data, our approach surpasses modern ones for dependent data, highlighting the effectiveness of our theoretically grounded loss functions for SSL in capturing spatio-temporal dependencies. Specifically, we outperform TS2Vec on the standard UEA and UCR benchmarks, with accuracy improvements of $4.17$\% and $2.08$\%, respectively. Furthermore, on the drought classification task, which involves complex spatio-temporal patterns, our method achieves a $7$\% higher ROC-AUC score.