Multi-scale Transformer Pyramid Networks for Multivariate Time Series Forecasting

TL;DR

This paper introduces MTPNet, a multi-scale transformer pyramid network that captures diverse temporal dependencies in multivariate time series forecasting, outperforming existing methods on benchmark datasets.

Contribution

The paper proposes a novel multi-scale transformer architecture and a dimension invariant embedding technique to better model complex seasonalities in multivariate time series.

Findings

MTPNet outperforms recent state-of-the-art methods on nine benchmark datasets.

The dimension invariant embedding effectively captures short-term dependencies.

Multi-scale representations improve forecasting accuracy across various datasets.

Abstract

Multivariate Time Series (MTS) forecasting involves modeling temporal dependencies within historical records. Transformers have demonstrated remarkable performance in MTS forecasting due to their capability to capture long-term dependencies. However, prior work has been confined to modeling temporal dependencies at either a fixed scale or multiple scales that exponentially increase (most with base 2). This limitation hinders their effectiveness in capturing diverse seasonalities, such as hourly and daily patterns. In this paper, we introduce a dimension invariant embedding technique that captures short-term temporal dependencies and projects MTS data into a higher-dimensional space, while preserving the dimensions of time steps and variables in MTS data. Furthermore, we present a novel Multi-scale Transformer Pyramid Network (MTPNet), specifically designed to effectively capture temporal dependencies at multiple unconstrained scales. The predictions are inferred from multi-scale latent representations obtained from transformers at various scales. Extensive experiments on nine benchmark datasets demonstrate that the proposed MTPNet outperforms recent state-of-the-art methods.