A Note on the Finite Sample Bias in Time Series Cross-Validation

TL;DR

This paper demonstrates that cross-validation for time series models, including VARs and martingale-like errors, inherently contains bias, challenging assumptions of bias-free model selection in such contexts.

Contribution

It provides a theoretical analysis showing that cross-validation bias persists in time series models, even with structures previously thought to mitigate it.

Findings

Cross-validation introduces bias in time series model selection.

Bias persists even with VAR and martingale error structures.

Implications for model evaluation practices in time series analysis.

Abstract

It is well known that model selection via cross validation can be biased for time series models. However, many researchers have argued that this bias does not apply when using cross-validation with vector autoregressions (VAR) or with time series models whose errors follow a martingale-like structure. I show that even under these circumstances, performing cross-validation on time series data will still generate bias in general.