eMargin: Revisiting Contrastive Learning with Margin-Based Separation

TL;DR

This paper investigates the impact of an adaptive margin in contrastive learning for time series, finding it improves clustering but not downstream classification performance.

Contribution

Introduces an adaptive margin (eMargin) into contrastive loss for time series, analyzing its effects on clustering and classification tasks.

Findings

eMargin improves unsupervised clustering metrics.

eMargin does not enhance downstream classification performance.

Adaptive margin's effect varies across evaluation metrics.

Abstract

We revisit previous contrastive learning frameworks to investigate the effect of introducing an adaptive margin into the contrastive loss function for time series representation learning. Specifically, we explore whether an adaptive margin (eMargin), adjusted based on a predefined similarity threshold, can improve the separation between adjacent but dissimilar time steps and subsequently lead to better performance in downstream tasks. Our study evaluates the impact of this modification on clustering performance and classification in three benchmark datasets. Our findings, however, indicate that achieving high scores on unsupervised clustering metrics does not necessarily imply that the learned embeddings are meaningful or effective in downstream tasks. To be specific, eMargin added to InfoNCE consistently outperforms state-of-the-art baselines in unsupervised clustering metrics, but struggles to achieve competitive results in downstream classification with linear probing. The source code is publicly available at https://github.com/sfi-norwai/eMargin.