Measuring Pre-training Data Quality without Labels for Time Series Foundation Models

TL;DR

This paper introduces a new label-free metric called contrastive accuracy to evaluate and select high-quality pre-training datasets for time series foundation models, improving their generalization across tasks.

Contribution

It proposes contrastive accuracy as a novel measure for assessing pre-training data quality without labels, aiding dataset selection for better foundation model performance.

Findings

Contrastive accuracy correlates with downstream task performance.

Using the measure can guide dataset selection for pre-training.

The approach enhances foundation model generalization.

Abstract

Recently, there has been a growing interest in time series foundation models that generalize across different downstream tasks. A key to strong foundation models is a diverse pre-training dataset, which is particularly challenging to collect for time series classification. In this work, we explore the performance of a contrastive-learning-based foundation model as a function of the data used for pre-training. We introduce contrastive accuracy, a new measure to evaluate the quality of the representation space learned by the foundation model. Our experiments reveal the positive correlation between the proposed measure and the accuracy of the model on a collection of downstream tasks. This suggests that the contrastive accuracy can serve as a criterion to search for time series datasets that can enhance the pre-training and improve thereby the foundation model's generalization.