GTM: A General Time-series Model for Enhanced Representation Learning of Time-Series Data

TL;DR

GTM introduces a novel frequency-domain attention mechanism and a hybrid pre-training strategy to improve time-series representation learning, enabling versatile adaptation to multiple tasks and outperforming state-of-the-art models.

Contribution

The paper presents GTM, the first generative-task-agnostic time-series model with a frequency-aware attention mechanism and a unified pre-training approach for enhanced generalization.

Findings

GTM outperforms SOTA models on various generative tasks.

GTM achieves strong classification results with minimal adaptation.

GTM's accuracy improves with larger model size and more pre-training data.

Abstract

Despite recent progress in time-series foundation models, challenges persist in improving representation learning and adapting to diverse downstream tasks. We introduce a General Time-series Model (GTM), which advances representation learning via a novel frequency-domain attention mechanism that captures time-granularity-aware features, an aspect underexplored in prior research. We further propose a novel pre-training strategy that unifies reconstruction and autoregressive objectives through a hybrid masking mechanism. Our pre-training strategy, combined with 2D positional encoding and span shuffling, enhances the robustness and generalization of representations. GTM is established as the first generative-task-agnostic model for time-series analysis, enabling seamless adaptation to various generative tasks without any task-specific modifications. Extensive experiments demonstrate that GTM consistently outperforms SOTA models on various generative tasks and achieves strong classification results with minimal adaptation. Furthermore, GTM exhibits clear scaling behavior, with accuracy improving as model size and pre-training data increase.