TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting

TL;DR

TEMPO introduces a prompt-based GPT-like framework for time series forecasting, leveraging decomposition and prompts to improve accuracy across diverse datasets and zero-shot scenarios.

Contribution

The paper presents TEMPO, a novel pre-trained transformer architecture for time series that incorporates domain-specific inductive biases and prompt design, enhancing generalization and performance.

Findings

Outperforms state-of-the-art methods on benchmark datasets in zero-shot settings.

Effective in handling unseen datasets and multi-modal inputs.

Demonstrates potential as a foundational model for time series forecasting.

Abstract

The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the-art results, the best-performing architectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demonstrated impressive performance via training one general-purpose model across various textual datasets. It is intriguing to explore whether GPT-type architectures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components; and (ii) introducing the design of prompts to facilitate distribution adaptation in different types of time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over state-of-the-art methods on zero shot setting for a number of time series benchmark datasets. This performance gain is observed not only in scenarios involving previously unseen datasets but also in scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute a foundational model-building framework.