TimesBERT: A BERT-Style Foundation Model for Time Series Understanding

TL;DR

TimesBERT introduces a BERT-style foundation model for time series understanding, capturing multi-granularity structures and achieving state-of-the-art results across various tasks by leveraging large-scale pre-training.

Contribution

The paper pioneers the adaptation of BERT-style architecture for time series understanding, incorporating a novel parallel functional token prediction task.

Findings

Achieves state-of-the-art performance on four downstream tasks.

Pre-trained on 260 billion time points across diverse domains.

Outperforms task-specific models and language pre-trained backbones.

Abstract

Time series analysis is crucial in diverse scenarios. Beyond forecasting, considerable real-world tasks are categorized into classification, imputation, and anomaly detection, underscoring different capabilities termed time series understanding in this paper. While GPT-style models have been positioned as foundation models for time series forecasting, the BERT-style architecture, which has made significant advances in natural language understanding, has not been fully unlocked for time series understanding, possibly attributed to the undesirable dropout of essential elements of BERT. In this paper, inspired by the shared multi-granularity structure between multivariate time series and multisentence documents, we design TimesBERT to learn generic representations of time series including temporal patterns and variate-centric characteristics. In addition to a natural adaptation of masked modeling, we propose a parallel task of functional token prediction to embody vital multi-granularity structures. Our model is pre-trained on 260 billion time points across diverse domains. Leveraging multi-granularity representations, TimesBERT achieves state-of-the-art performance across four typical downstream understanding tasks, outperforming task-specific models and language pre-trained backbones, positioning it as a versatile foundation model for time series understanding.