Dateformer: Time-modeling Transformer for Longer-term Series Forecasting

TL;DR

Dateformer introduces a novel patch-wise Transformer approach for long-term time series forecasting, leveraging global information through time representations to significantly improve accuracy and forecast range.

Contribution

The paper proposes a new patch-wise Transformer model that utilizes global training data via time representations, overcoming the limitations of narrow lookback windows.

Findings

Achieves 33.6% relative accuracy improvement on 7 datasets

Extends forecast horizon to half a year

Outperforms existing models in long-term series forecasting

Abstract

Transformers have demonstrated impressive strength in long-term series forecasting. Existing prediction research mostly focused on mapping past short sub-series (lookback window) to future series (forecast window). The longer training dataset time series will be discarded, once training is completed. Models can merely rely on lookback window information for inference, which impedes models from analyzing time series from a global perspective. And these windows used by Transformers are quite narrow because they must model each time-step therein. Under this point-wise processing style, broadening windows will rapidly exhaust their model capacity. This, for fine-grained time series, leads to a bottleneck in information input and prediction output, which is mortal to long-term series forecasting. To overcome the barrier, we propose a brand-new methodology to utilize Transformer for time series forecasting. Specifically, we split time series into patches by day and reform point-wise to patch-wise processing, which considerably enhances the information input and output of Transformers. To further help models leverage the whole training set's global information during inference, we distill the information, store it in time representations, and replace series with time representations as the main modeling entities. Our designed time-modeling Transformer -- Dateformer yields state-of-the-art accuracy on 7 real-world datasets with a 33.6\% relative improvement and extends the maximum forecast range to half-year.