HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting

TL;DR

The paper introduces HDT, a hierarchical discrete transformer that models high-dimensional multivariate time series using discrete token representations, enabling scalable and long-term forecasting with improved accuracy.

Contribution

It proposes a novel hierarchical discrete transformer framework that models multivariate time series as discrete tokens, addressing scalability and long-term forecasting challenges.

Findings

HDT outperforms existing models on five MTS datasets.

The discrete token approach enables faster inference.

Hierarchical modeling extends forecasting length effectively.

Abstract

Generative models have gained significant attention in multivariate time series forecasting (MTS), particularly due to their ability to generate high-fidelity samples. Forecasting the probability distribution of multivariate time series is a challenging yet practical task. Although some recent attempts have been made to handle this task, two major challenges persist: 1) some existing generative methods underperform in high-dimensional multivariate time series forecasting, which is hard to scale to higher dimensions; 2) the inherent high-dimensional multivariate attributes constrain the forecasting lengths of existing generative models. In this paper, we point out that discrete token representations can model high-dimensional MTS with faster inference time, and forecasting the target with long-term trends of itself can extend the forecasting length with high accuracy. Motivated by this, we propose a vector quantized framework called Hierarchical Discrete Transformer (HDT) that models time series into discrete token representations with l2 normalization enhanced vector quantized strategy, in which we transform the MTS forecasting into discrete tokens generation. To address the limitations of generative models in long-term forecasting, we propose a hierarchical discrete Transformer. This model captures the discrete long-term trend of the target at the low level and leverages this trend as a condition to generate the discrete representation of the target at the high level that introduces the features of the target itself to extend the forecasting length in high-dimensional MTS. Extensive experiments on five popular MTS datasets verify the effectiveness of our proposed method.