Quantifying the Pre-training Dividend: Generative versus Latent Self-Supervised Learning for Time Series Foundation Models

TL;DR

This paper evaluates the benefits of self-supervised learning for time series, comparing generative and latent approaches, and finds that SSL provides significant gains for some tasks but not others, influenced by task-specific signal resolution needs.

Contribution

It introduces a controlled framework for quantifying the pre-training dividend in time series SSL and compares generative and latent methods with novel adaptations.

Findings

SSL yields up to 375% improvements in anomaly detection and classification.

Representation utility depends on task-specific signal resolution requirements.

Representation quality saturates at moderate model depths, independent of data source.

Abstract

The success of self-supervised learning (SSL) in vision and NLP has motivated its rapid adoption for time series. However, research has focused primarily on Generative paradigms and forecasting tasks, leaving the broader utility of learned representations unquantified. We establish a controlled framework to evaluate the "pre-training dividend": the value added by SSL across diverse temporal tasks. We systematically compare Generative paradigms against Latent Alignment architectures, introducing adaptations of LeJEPA and DINO for time series. These adaptations utilize Discrete Wavelet Transform (DWT) augmentations to enforce invariance to local fluctuations. Our analysis reveals that the pre-training dividend is highly asymmetric: SSL yields gains of up to 375% for anomaly detection and classification, yet remains marginal for forecasting. We demonstrate that representational utility is non-universal, governed by a precision-invariance trade-off where the specific signal resolution required by the task must align with the objective. Finally, we show that representation quality is largely independent of data origin and saturates at moderate architectural depths, suggesting a path to scaling via massive synthetic generation. Our code is available at: https://github.com/noammajor/Models