Uncertainty Intervals for Prediction Errors in Time Series Forecasting

TL;DR

This paper develops statistically valid uncertainty intervals for prediction errors in time series forecasting, addressing limitations of existing methods and proposing a new quantile-based approach with theoretical guarantees.

Contribution

It introduces the QFCV method for constructing asymptotically valid prediction intervals for stochastic test errors in time series, with extensions for non-stationary data.

Findings

QFCV intervals have asymptotic coverage guarantees.

Naive CLT-based intervals are statistically invalid.

QFCV outperforms traditional methods in simulations and real data.

Abstract

Inference for prediction errors is critical in time series forecasting pipelines. However, providing statistically meaningful uncertainty intervals for prediction errors remains relatively under-explored. Practitioners often resort to forward cross-validation (FCV) for obtaining point estimators and constructing confidence intervals based on the Central Limit Theorem (CLT). The naive version assumes independence, a condition that is usually invalid due to time correlation. These approaches lack statistical interpretations and theoretical justifications even under stationarity. This paper systematically investigates uncertainty intervals for prediction errors in time series forecasting. We first distinguish two key inferential targets: the stochastic test error over near future data points, and the expected test error as the expectation of the former. The stochastic test error is often more relevant in applications needing to quantify uncertainty over individual time series instances. To construct prediction intervals for the stochastic test error, we propose the quantile-based forward cross-validation (QFCV) method. Under an ergodicity assumption, QFCV intervals have asymptotically valid coverage and are shorter than marginal empirical quantiles. In addition, we also illustrate why naive CLT-based FCV intervals fail to provide valid uncertainty intervals, even with certain corrections. For non-stationary time series, we further provide rolling intervals by combining QFCV with adaptive conformal prediction to give time-average coverage guarantees. Overall, we advocate the use of QFCV procedures and demonstrate their coverage and efficiency through simulations and real data examples.