Model Selection for Time Series Forecasting: Empirical Analysis of Different Estimators

TL;DR

This paper empirically compares various estimators for model selection in time series forecasting, revealing low accuracy in selecting the best model and quantifying the associated performance loss.

Contribution

It provides an empirical analysis of estimator effectiveness for model selection in time series forecasting, highlighting factors influencing their performance.

Findings

Estimators have low accuracy in selecting the best model.

Performance loss due to model selection ranges from 1.2% to 2.3%.

Sample size significantly affects estimator performance.

Abstract

Evaluating predictive models is a crucial task in predictive analytics. This process is especially challenging with time series data where the observations show temporal dependencies. Several studies have analysed how different performance estimation methods compare with each other for approximating the true loss incurred by a given forecasting model. However, these studies do not address how the estimators behave for model selection: the ability to select the best solution among a set of alternatives. We address this issue and compare a set of estimation methods for model selection in time series forecasting tasks. We attempt to answer two main questions: (i) how often is the best possible model selected by the estimators; and (ii) what is the performance loss when it does not. We empirically found that the accuracy of the estimators for selecting the best solution is low, and the overall forecasting performance loss associated with the model selection process ranges from 1.2% to 2.3%. We also discovered that some factors, such as the sample size, are important in the relative performance of the estimators.