Scalable Transformer for High Dimensional Multivariate Time Series Forecasting

TL;DR

This paper introduces STHD, a scalable Transformer model designed to improve high-dimensional multivariate time series forecasting by addressing noise and training challenges in channel-dependent models.

Contribution

The paper proposes STHD, a novel scalable Transformer architecture with relation matrix sparsity, a reindexing training strategy, and 2-D input handling to effectively forecast high-dimensional MTS data.

Findings

STHD outperforms existing models on three high-dimensional datasets.

The relation matrix sparsity reduces noise and memory usage.

ReIndex enhances training flexibility and data diversity.

Abstract

Deep models for Multivariate Time Series (MTS) forecasting have recently demonstrated significant success. Channel-dependent models capture complex dependencies that channel-independent models cannot capture. However, the number of channels in real-world applications outpaces the capabilities of existing channel-dependent models, and contrary to common expectations, some models underperform the channel-independent models in handling high-dimensional data, which raises questions about the performance of channel-dependent models. To address this, our study first investigates the reasons behind the suboptimal performance of these channel-dependent models on high-dimensional MTS data. Our analysis reveals that two primary issues lie in the introduced noise from unrelated series that increases the difficulty of capturing the crucial inter-channel dependencies, and challenges in training strategies due to high-dimensional data. To address these issues, we propose STHD, the Scalable Transformer for High-Dimensional Multivariate Time Series Forecasting. STHD has three components: a) Relation Matrix Sparsity that limits the noise introduced and alleviates the memory issue; b) ReIndex applied as a training strategy to enable a more flexible batch size setting and increase the diversity of training data; and c) Transformer that handles 2-D inputs and captures channel dependencies. These components jointly enable STHD to manage the high-dimensional MTS while maintaining computational feasibility. Furthermore, experimental results show STHD's considerable improvement on three high-dimensional datasets: Crime-Chicago, Wiki-People, and Traffic. The source code and dataset are publicly available https://github.com/xinzzzhou/ScalableTransformer4HighDimensionMTSF.git.