Scaling-laws for Large Time-series Models

TL;DR

This paper demonstrates that large-scale decoder-only transformer models for time series forecasting follow similar scaling laws to language models, with performance improving predictably as model size, data, and compute increase.

Contribution

It establishes for the first time power-law scaling laws for large time series transformer models across multiple parameters and datasets.

Findings

Scaling laws hold for time series transformers similar to language models.

Architectural details have minimal impact on scaling behavior.

Performance improves predictably with model size, data, and compute.

Abstract

Scaling laws for large language models (LLMs) have provided useful guidance in training ever larger models for predictable performance gains. Time series forecasting shares a similar sequential structure to language, and is amenable to large-scale transformer architectures. Here we show that foundational decoder-only time series transformer models exhibit analogous scaling-behavior to LLMs, with architectural details (aspect ratio and number of heads) having a minimal effect over broad ranges. We assemble a large corpus of heterogenous time series data on which to train, and establish for the first time power-law scaling with parameter count, dataset size, and training compute, spanning five orders of magnitude.