MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention

TL;DR

This paper introduces MQTransformer, a neural forecasting model that leverages Transformer-inspired attention mechanisms and novel positional encoding to improve probabilistic demand prediction accuracy and reduce forecast variability.

Contribution

It proposes a new decoder-encoder attention, context-dependent positional encoding, and a decoder-self attention scheme, advancing neural forecasting methods.

Findings

Improved forecasting accuracy with context-aware attention.

Reduced forecast variability through self-attention mechanisms.

Enhanced modeling of seasonality and holiday effects.

Abstract

Recent advances in neural forecasting have produced major improvements in accuracy for probabilistic demand prediction. In this work, we propose novel improvements to the current state of the art by incorporating changes inspired by recent advances in Transformer architectures for Natural Language Processing. We develop a novel decoder-encoder attention for context-alignment, improving forecasting accuracy by allowing the network to study its own history based on the context for which it is producing a forecast. We also present a novel positional encoding that allows the neural network to learn context-dependent seasonality functions as well as arbitrary holiday distances. Finally we show that the current state of the art MQ-Forecaster (Wen et al., 2017) models display excess variability by failing to leverage previous errors in the forecast to improve accuracy. We propose a novel decoder-self attention scheme for forecasting that produces significant improvements in the excess variation of the forecast.