Evaluating financial tail risk forecasts: Testing Equal Predictive Ability

TL;DR

This paper evaluates the finite sample performance of the Diebold-Mariano and Hansen et al. tests in assessing financial tail risk forecasts like VaR and ES, highlighting limitations and conditions affecting their power.

Contribution

It provides a comprehensive simulation study on the finite sample properties of key tests for tail risk forecast evaluation, emphasizing the impact of asymmetric loss functions and data characteristics.

Findings

Tests have low power for extreme tail underestimation.

Finite sample properties improve with higher quantiles and larger samples.

Skewed test statistics and type III errors occur at small quantiles and short sample sizes.

Abstract

This paper provides comprehensive simulation results on the finite sample properties of the Diebold-Mariano (DM) test by Diebold and Mariano (1995) and the model confidence set (MCS) testing procedure by Hansen et al. (2011) applied to the asymmetric loss functions specific to financial tail risk forecasts, such as Value-at-Risk (VaR) and Expected Shortfall (ES). We focus on statistical loss functions that are strictly consistent in the sense of Gneiting (2011a). We find that the tests show little power against models that underestimate the tail risk at the most extreme quantile levels, while the finite sample properties generally improve with the quantile level and the out-of-sample size. For the small quantile levels and out-of-sample sizes of up to two years, we observe heavily skewed test statistics and non-negligible type III errors, which implies that researchers should be cautious about using standard normal or bootstrapped critical values. We demonstrate both empirically and theoretically how these unfavorable finite sample results relate to the asymmetric loss functions and the time varying volatility inherent in financial return data.