Forecast Encompassing Tests for the Expected Shortfall

TL;DR

This paper develops new statistical tests to evaluate the accuracy of Expected Shortfall forecasts, a key risk measure in banking regulation, using robust theory and simulation studies.

Contribution

It introduces novel forecast encompassing tests for Expected Shortfall, incorporating joint loss functions and robust asymptotic theory, with applications to financial asset forecasting.

Findings

Tests perform well in finite samples

Forecast combination methods improve risk prediction

Applicable to various financial assets

Abstract

We introduce new forecast encompassing tests for the risk measure Expected Shortfall (ES). The ES currently receives much attention through its introduction into the Basel III Accords, which stipulate its use as the primary market risk measure for the international banking regulation. We utilize joint loss functions for the pair ES and Value at Risk to set up three ES encompassing test variants. The tests are built on misspecification robust asymptotic theory and we investigate the finite sample properties of the tests in an extensive simulation study. We use the encompassing tests to illustrate the potential of forecast combination methods for different financial assets.

Tables

Table 1. Table 1 : Empirical Sizes of the Forecast Encompassing Tests.

		${ℍ}_0^{\left(1\right)}$					${ℍ}_0^{\left(2\right)}$
		Str ES	Aux ES	VaR ES	VaR		Str ES	Aux ES	VaR ES	VaR
$n$		GARCH
$500$		15.25	15.20	18.35	22.75		14.40	14.65	18.80	22.50
$1000$		11.55	11.10	15.60	20.10		12.30	12.70	17.80	22.85
$2500$		11.45	11.55	16.35	18.80		11.00	11.25	14.60	17.55
$5000$		10.05	10.25	13.10	15.35		9.75	10.15	13.90	15.75

Table 2. Table 2 : Encompassing Test Results for the IBM Stock, the DAX 30 and the S&P 500 Indices.

		Strict ES					Aux ES					Joint VaR ES					VaR
Model		NR	E1	E2	C		NR	E1	E2	C		NR	E1	E2	C		NR	E1	E2	C
		IBM
HS			57		43			57		43			43		57			43		57
RM			57		43			57		43			43		57		14	43		43
GJR			57		43			57		43			43		57		14	29		57
GAS			43		57			43		57			29		71			29		71
G1F		14	29	43	14		14	29	43	14			14		86			14		86
G2F		14	29	57			14	29	57				29	57	14			29	43	29
ASES		14		86			14		86			14		57	29		14		57	29
SAVES		14		86			14		86			14		86			14		86

Equations

$$\displaystyle\mathbb{E}\left[\rho\big{(}Y_{t+1},\hat{f}_{1,t}\big{)}\right]\leq\mathbb{E}\left[\rho\big{(}Y_{t+1},\theta_{1}\hat{f}_{1,t}+\theta_{2}\hat{f}_{2,t}\big{)}\right],$$

$$\displaystyle\big{(}\theta^{\ast}_{1},\theta^{\ast}_{2}\big{)}=\underset{(\theta_{1},\theta_{2})\in\Theta}{\operatorname{arg\,min}}\;\mathbb{E}\left[\rho\big{(}Y_{t+1},\theta_{1}\hat{f}_{1,t}+\theta_{2}\hat{f}_{2,t}\big{)}\right].$$

$$\displaystyle\mathbb{E}\left[\rho\big{(}Y_{t+1},\hat{f}_{1,t}\big{)}\right]\geq\mathbb{E}\left[\rho\big{(}Y_{t+1},\theta^{\ast}_{1}\hat{f}_{1,t}+\theta^{\ast}_{2}\hat{f}_{2,t}\big{)}\right].$$

$$\displaystyle\mathbb{E}\left[\rho\big{(}Y_{t+1},\hat{f}_{1,t}\big{)}\right]=\mathbb{E}\left[\rho\big{(}Y_{t+1},\theta^{\ast}_{1}\hat{f}_{1,t}+\theta^{\ast}_{2}\hat{f}_{2,t}\big{)}\right],$$

$$g:\mathfrak{F}\times \Theta \to \mathbb{R},(\widehat{{\mathbf{f}}_t},\theta )\mapsto g(\widehat{{\mathbf{f}}_t},\theta ),$$

$$\displaystyle\mathbb{E}\left[\rho\big{(}Y_{t+1},\hat{f}_{1,t}\big{)}\right]=\mathbb{E}\left[\rho\big{(}Y_{t+1},g(\hat{\boldsymbol{f}_{t}},\theta^{\ast})\big{)}\right],$$

$$E{S}_{t,\alpha }(Y_{t+1})={\mathbb{E}}_t\left[Y_{t+1}|Y_{t+1}\le Q_{t,\alpha }(Y_{t+1})\right],$$

$$\begin{array}{r}\rho (Y,q_{\alpha },e_{\alpha })=-\frac{1}{e_{\alpha }}\left(e_{\alpha }-q_{\alpha }+\frac{(q_{\alpha }-Y)\text{mathds}{1}_{\{Y\le q_{\alpha }\}}}{\alpha }\right)+\log (-e_{\alpha }),\end{array}$$

$$g^q:\mathfrak{Q}\times {\Theta }^{\beta }\to \mathbb{R}$$

$$g^e:\mathfrak{E}\times {\Theta }^{\eta }\to \mathbb{R}$$

$$\displaystyle\theta^{\ast}=(\beta^{\ast},\eta^{\ast})=\underset{(\beta,\eta)\in\Theta}{\operatorname{arg\,min}}\;\mathbb{E}\left[\rho\big{(}Y_{t+1},g^{q}(\hat{\boldsymbol{q}}_{t},\beta),g^{e}(\hat{\boldsymbol{e}}_{t},\eta)\big{)}\right].$$

$$\displaystyle\mathbb{E}\left[\rho\big{(}Y_{t+1},\hat{q}_{1,t},\hat{e}_{1,t}\big{)}\right]=\mathbb{E}\left[\rho\big{(}Y_{t+1},g^{q}(\hat{\boldsymbol{q}}_{t},\beta^{\ast}),g^{e}(\hat{\boldsymbol{e}}_{t},\eta^{\ast})\big{)}\right],$$

$$\begin{array}{r}{Y}_{t+1}=g^q({\hat{\mathbf{q}}}_t,\beta )+u_{t+1}^q,\text{ and }{Y}_{t+1}=g^e({\hat{\mathbf{e}}}_t,\eta )+u_{t+1}^e,\end{array}$$

$$\displaystyle\mathbb{E}\left[\rho\big{(}Y_{t+1},g^{q}(\hat{\boldsymbol{q}}_{t},\beta^{\ast}),\hat{e}_{1,t}\big{)}\right]=\mathbb{E}\left[\rho\big{(}Y_{t+1},g^{q}(\hat{\boldsymbol{q}}_{t},\beta^{\ast}),g^{e}(\hat{\boldsymbol{e}}_{t},\eta^{\ast})\big{)}\right],$$

$$\displaystyle\theta^{\ast}=(\beta^{\ast},\eta^{\ast})=\underset{(\beta,\eta)\in\Theta}{\operatorname{arg\,min}}\;\mathbb{E}\left[\rho\big{(}Y_{t+1},g^{q}(\hat{\boldsymbol{e}}_{t},\beta),g^{e}(\hat{\boldsymbol{e}}_{t},\eta)\big{)}\right].$$

$$\displaystyle\mathbb{E}\left[\rho\big{(}Y_{t+1},g^{q}(\hat{\boldsymbol{e}}_{t},\beta^{\ast}),\hat{e}_{1,t}\big{)}\right]=\mathbb{E}\left[\rho\big{(}Y_{t+1},g^{q}(\hat{\boldsymbol{e}}_{t},\beta^{\ast}),g^{e}(\hat{\boldsymbol{e}}_{t},\eta^{\ast})\big{)}\right],$$

$$\begin{array}{r}{Y}_{t+1}=g^q({\hat{\mathbf{e}}}_t,\beta )+u_{t+1}^q,\text{ and }{Y}_{t+1}=g^e({\hat{\mathbf{e}}}_t,\eta )+u_{t+1}^e,\end{array}$$

$$\displaystyle\hat{\theta}_{n}=\underset{\theta\in\Theta}{\operatorname*{arg\,min}}\,Q_{n}(\theta),\quad\text{ where }\quad Q_{n}(\theta)=\frac{1}{n}\sum_{t=m}^{T-1}\rho\big{(}Y_{t+1},g^{q}_{t}(\beta),g^{e}_{t}(\eta)\big{)},$$

$$\displaystyle\theta^{\ast}_{n}=\underset{\theta\in\Theta}{\operatorname*{arg\,min}}\,Q_{n}^{0}(\theta),\quad\text{ where }\quad Q_{n}^{0}(\theta)=\frac{1}{n}\sum_{t=m}^{T-1}\mathbb{E}\left[\rho\big{(}Y_{t+1},g^{q}_{t}(\beta),g^{e}_{t}(\eta)\big{)}\right].$$

$$\displaystyle\psi\big{(}Y_{t+1},g^{q}_{t}(\beta),g^{e}_{t}(\eta)\big{)}=\begin{pmatrix}-\frac{\nabla g^{q}_{t}(\beta)}{\alpha g^{e}_{t}(\eta)}\left(\mathds{1}_{\{Y_{t+1}\leq g^{q}_{t}(\beta)\}}-\alpha\right)\\ \frac{\nabla g^{e}_{t}(\eta)}{g^{e}_{t}(\eta)^{2}}\left(g^{e}_{t}(\eta)-g^{q}_{t}(\beta)+\frac{1}{\alpha}(g^{q}_{t}(\beta)-Y_{t+1})\mathds{1}_{\{Y_{t+1}\leq g^{q}_{t}(\beta)\}}\right)\end{pmatrix}.$$

$$\displaystyle\Omega_{n}^{-1/2}(\theta^{\ast}_{n})\sqrt{n}\big{(}\hat{\theta}_{n}-\theta^{\ast}_{n}\big{)}\stackrel{{\scriptstyle d}}{{\longrightarrow}}\mathcal{N}(0,I_{k}),$$

$${\Lambda }_{n,qq}({\theta }_n^{\ast })$$

$${\Lambda }_{n,qe}({\theta }_n^{\ast })$$

$${\Lambda }_{n,ee}({\theta }_n^{\ast })$$

$$\left(g_t^e({\eta }_n^{\ast })-g_t^q({\beta }_n^{\ast })+\frac{1}{\alpha }(g_t^q({\beta }_n^{\ast })-Y_{t+1})\text{mathds}{1}_{\{Y_{t+1}\le g_t^q({\beta }_n^{\ast })\}}\right)],$$

$$Z_{R,n}$$

$$Z_{R,n}\stackrel{d}{⟶}{\chi }_s^2.$$

$$\begin{array}{rlrlrl}& \text{Joint}:& & {\mathbb{H}}_0^{(1)}:({\beta }_2^{\ast },{\beta }_3^{\ast },{\eta }_2^{\ast },{\eta }_3^{\ast })=(1,0,1,0),& & {\mathbb{H}}_0^{(2)}:({\beta }_2^{\ast },{\beta }_3^{\ast },{\eta }_2^{\ast },{\eta }_3^{\ast })=(0,1,0,1),\\ & \text{Str \& Aux}:& & {\mathbb{H}}_0^{(1)}:({\eta }_2^{\ast },{\eta }_3^{\ast })=(1,0),& & {\mathbb{H}}_0^{(2)}:({\eta }_2^{\ast },{\eta }_3^{\ast })=(0,1),\\ & \text{VaR}:& & {\mathbb{H}}_0^{(1)}:({\beta }_2^{\ast },{\beta }_3^{\ast })=(1,0),& & {\mathbb{H}}_0^{(2)}:({\beta }_2^{\ast },{\beta }_3^{\ast })=(0,1).\end{array}$$

$${\hat{\sigma }}_{1,t}^2$$

$${\hat{\sigma }}_{2,t}^2$$

$${\hat{q}}_{1,t}$$

$${\hat{\kappa }}_t$$

$$\left(\begin{array}{c}{\hat{q}}_{2,t}\\ {\hat{e}}_{2,t}\end{array}\right)=\left(\begin{array}{c}-0.009\\ -0.010\end{array}\right)+\left(\begin{array}{cc}0.993& 0\\ 0& 0.994\end{array}\right)\left(\begin{array}{c}{\hat{q}}_{2,t-1}\\ {\hat{e}}_{2,t-1}\end{array}\right)+\left(\begin{array}{cc}-0.358& -0.351\\ -0.003& -0.003\end{array}\right){\lambda }_t,$$

Full text

Keywords: evaluating forecasts, combining forecasts, loss function, model selection, statistical tests

1 Introduction

Through the recent introduction of Expected Shortfall (ES) as the primary market risk measure for the international banking regulation in the Basel III Accords (Basel Committee, , 2016, 2017), there is a great demand for reliable methods for evaluating and comparing the predictive ability of competing ES forecasts. The ES at probability level $\alpha\in(0,1)$ is defined as the expectation of the returns smaller than the respective $\alpha$-quantile (the Value at Risk, VaR), where $\alpha$ is usually chosen to be 2.5% as proposed by the Basel Accords. The ES is replacing the VaR in the banking regulation as it overcomes several shortcomings of the latter such as being not coherent and its inability to capture tail risks beyond the $\alpha$-quantile (Artzner et al., , 1999; Danielsson et al., , 2001; Basel Committee, , 2013). While the empirical properties favor the ES over the VaR as a risk measure, the ES lacks elicitability, which implies that no strictly consistent loss functions exist. The non-elicitability of the ES is overcome by considering the pair VaR and ES which are jointly elicitable, i.e. there exist joint loss functions for the VaR and the ES (Fissler and Ziegel, , 2016). This discovery triggered a rapidly growing branch of literature in developing forecasting methods and forecast evaluation techniques for the ES, see Patton et al., (2019), Dimitriadis and Bayer, (2019), Bayer and Dimitriadis, (2020), Taylor, (2019), Barendse, (2020), Fissler et al., (2016) and Nolde and Ziegel, (2017) among others.

A desirable tool for the comparison of ES forecasts are encompassing tests, which however build upon the existence of strictly consistent loss functions. Given two competing forecasts A and B, forecast encompassing tests the null hypothesis that forecast A performs not worse than any (linear) combination of these forecasts. This is carried out by testing whether the optimal combination weight of forecast B deviates significantly from zero.222For the classical theory on forecast encompassing see Hendry and Richard, (1982), Mizon and Richard, (1986), Diebold, (1989), Ericsson, (1993), Harvey et al., (1998), Clark and McCracken, (2001), Giacomini and Komunjer, (2005), Newbold and Harvey, (2007) and Clements and Harvey, (2009) among others. This null hypothesis allows for the convenient interpretation that forecast B does not add any information to forecast A and thus, forecast A is superior to forecast B. The existence of appropriate loss functions is inevitable for encompassing tests for two reasons. First, the superior performance of competing forecasts is defined in the statistical sense by using strictly consistent loss functions. Second, loss and identification functions are crucial for M- or GMM-estimation of the optimal forecast combination weights through an appropriate regression framework for the risk measure under consideration.

In this paper, we introduce novel encompassing tests for the ES based on the joint loss functions for the VaR and ES developed in Fissler and Ziegel, (2016). We introduce the following three test variants for the ES. First, we propose to jointly test forecast encompassing for the VaR and ES, henceforth denoted the joint VaR and ES encompassing test. We introduce a second test variant, denoted the auxiliary ES encompassing test, which estimates the optimal combination weights for the vector of the VaR and ES, however, only tests the parameters associated with the ES. While incorporating both, VaR and ES forecasts, this variant only tests encompassing of the ES forecasts. The third variant overcomes the tests’ dependence on VaR forecasts and tests encompassing of competing ES forecasts stand-alone, which comes at the cost of a potential model misspecification. We henceforth call this test the strict ES encompassing test. This variant is particularly relevant due to the current set of rules established by the Basel Committee of Banking Supervision, which only imposes the financial institutions to report ES forecasts (Basel Committee, , 2016, 2017). Only this test variant can be applied in situations where the person evaluating the forecasts merely has forecasts for the ES at hand. However, in situations where both, the VaR and ES forecasts (stemming from the same model or forecasting procedure) are available, application of the joint or auxiliary tests is generally recommended.

We implement the encompassing tests through M-estimation of the optimal combination weights (Patton et al., , 2019; Dimitriadis and Bayer, , 2019) and in an environment with asymptotically non-vanishing estimation uncertainty of the forecasting procedures (Giacomini and Komunjer, , 2005; Giacomini and White, , 2006). As the strict ES encompassing test is potentially subject to model misspecification, we derive the asymptotic distribution of the test statistics in a general setting which allows for misspecified models. This generalizes the asymptotic theory of Patton et al., (2019), Dimitriadis and Bayer, (2019) and Bayer and Dimitriadis, (2020) to potentially misspecified (and nonlinear) models. We base the Wald test statistics of the encompassing tests on a misspecification-robust covariance estimator. Our implementation further introduces a link or combination function which captures the different linear and nonlinear forecast combination methods in the existing encompassing testing literature, see Clements and Harvey, (2009) and Clements and Harvey, (2010) among others.

We analyze the finite sample behavior of our encompassing tests and the effect of the potential model misspecification in an extensive simulation study using models from various model classes associated with the ES. For this, we consider classical GARCH models, the GAS (generalized auto-regressive score) models with time-varying higher moments of Creal et al., (2013), the GAS models for the VaR and ES of Patton et al., (2019) and the ES-CAViaR models of Taylor, (2019). Data stemming from the latter three model classes induces some model misspecification for the strict ES encompassing test, which allows us to evaluate the effect the misspecification has on our tests. We find that all tests exhibit approximately correct size and good power properties for all considered simulations. This also holds for the strict ES encompassing test which demonstrates that this test is robust to the degree of model misspecification we usually encounter in financial applications.

Tests for forecast encompassing are commonly used to establish a theoretical basis for forecast combinations in cases when encompassing is rejected for both forecasts (Clements and Harvey, , 2009; Newbold and Harvey, , 2007; Giacomini and Komunjer, , 2005). This implies that neither of the forecasts stand-alone performs as good as an optimal forecast combination, which indicates that a forecast combination incorporates more information than the individual forecasts. Giacomini and Komunjer, (2005), Timmermann, (2006), Halbleib and Pohlmeier, (2012) and Taylor, (2020) advocate general forecast combination methods for multiple reasons and particularly for risk measures with small probability levels, as it is customary for the VaR and the ES.

We apply our encompassing tests to ES forecasts from classical GARCH and GAS models, but also from the recently developed dynamic ES models of Taylor, (2019) and Patton et al., (2019) for daily returns of the IBM stock, the S$\&$P 500 and the DAX 30 indices. The test results imply that for the IBM stock, forecast combination methods outperform the stand-alone forecasting models in many instances. In comparison, this pattern seems to be less pronounced for the S$\&$P 500 and the DAX 30 indices, which are already well diversified through their versatile composition. Thus, classical diversification gains (Timmermann, , 2006) of forecast combination methods might be less pronounced for stock indices. The two ES based test variants exhibits very similar results, which further indicates that the strict ES test is robust against potential misspecifications in financial settings.

The classical idea of forecast encompassing goes back to Hendry and Richard, (1982), Chong and Hendry, (1986) and Mizon and Richard, (1986) and is developed for mean forecasts under the squared loss function. Broad reviews on encompassing testing are provided e.g. by Newbold and Harvey, (2007) and Clements and Harvey, (2009). Harvey and Newbold, (2000) extend the encompassing technique which classically focuses on two competing forecasts to encompassing of multiple forecasts. Giacomini and Komunjer, (2005) develop (conditional) encompassing of quantile forecasts and focus on encompassing tests for methods instead of models. Clements and Harvey, (2010) generalize encompassing tests to probabilistic forecasts by relying on strictly consistent scoring rules. Giacomini and Komunjer, (2005) and Clements and Harvey, (2010) investigate extensions of encompassing to more complicated functionals of the conditional distribution. Our work pursues this path by developing encompassing tests for the ES as a prominent example of higher-order elicitable functionals where only joint loss functions for vector-valued functionals are available. Our testing approach can be adapted to further higher-order elicitable functionals such as the pair mean, variance and the Range Value at Risk (Cont et al., , 2010; Embrechts et al., , 2018; Fissler and Ziegel, , 2019).

The rest of the paper is organized as follows. In Section 2, we introduce encompassing tests for the ES and derive the asymptotic distribution of the associated test statistics under model misspecification. Section 3 presents an extensive simulation study analyzing the size and power properties of our tests. In Section 4, we apply the testing procedure to daily financial returns of the IBM stock and the S&P 500 and DAX 30 indices and Section 5 concludes. All proofs are deferred to Appendix A. Technical details of the proofs and additional results are provided in the supplementary material.

2 Theory

We consider a stochastic process $Z=\left\{Z_{t}:\Omega\to\mathbb{R}^{l+1},l\in\mathbb{N},t=1,\dots,T\right\}$, which is defined on some common and complete probability space $(\Omega,\mathcal{F},\mathbb{P})$, where $\mathcal{F}=\left\{\mathcal{F}_{t},t=1,\dots,T\right\}$ and $\mathcal{F}_{t}=\sigma\left\{Z_{s},s\leq t\right\}$. We partition the stochastic process as $Z_{t}=(Y_{t},X_{t})$, where $Y_{t}:\Omega\to\mathbb{R}$ is an absolutely continuous random variable of interest and $X_{t}:\Omega\to\mathbb{R}^{l}$ is a vector of explanatory variables. We denote the conditional distribution of $Y_{t+1}$ given the information set $\mathcal{F}_{t}$ by $F_{t}$. Accordingly, $\mathbb{E}_{t}$, $\operatorname{Var}_{t}$ and $h_{t}$ denote the expectation, variance and density corresponding to $F_{t}$. Following Giacomini and Komunjer, (2005), we consider ($\mathcal{F}_{t}$-measurable) one-step ahead forecasts, henceforth denoted by $\hat{f}_{t},\hat{q}_{t}$ and $\hat{e}_{t}$, which are generated by a function f\big{(}\gamma_{t,m},Z_{t},Z_{t-1},\dots\big{)}, which is fixed over time. For this, $\gamma_{t,m}$ denotes the (estimated) model parameters at time $t$ or alternatively the semi- or non-parametric estimator used in the construction of the forecasts. This construction allows for both, fixed forecasting schemes, where the model parameters $\gamma_{t,m}$ are only estimated once, and rolling window forecasting schemes, where the parameters $\gamma_{t,m}$ are re-estimated in each step. We denote general competing forecasts by $\hat{\boldsymbol{f}_{t}}=(\hat{f}_{1,t},\hat{f}_{2,t})$, specific VaR (quantile) forecasts by $\hat{\boldsymbol{q}}_{t}=(\hat{q}_{1,t},\hat{q}_{2,t})$ and ES forecasts by $\hat{\boldsymbol{e}}_{t}=(\hat{e}_{1,t},\hat{e}_{2,t})$.

In the context of evaluating point forecasts, an important property of risk measures (or more general statistical functionals) is elicitability (Gneiting, , 2011). Elicitability means that there exist strictly consistent loss functions, i.e. loss functions $\rho(Y,f)$ depending on the random variable $Y\sim F$ and the issued forecast $f$, whose expectation $\mathbb{E}\left[\rho(Y,\cdot)\right]$ is uniquely minimized by the true risk measure $\Gamma(F)$. Using such a loss function, one can assess the quality of issued forecasts by comparing their average losses induced by the realizations of the predicted variable. Evaluating forecasts through strictly consistent loss functions has the desired impact that it incentivizes financial institutions to truthfully report their correct forecasts (Gneiting, , 2011; Fissler et al., , 2016). As a direct consequence, the literature on tests for forecast comparison and forecast rationality evolves around the associated loss functions, see Mizon and Richard, (1986), Diebold and Mariano, (1995), Elliott et al., (2005), Giacomini and Komunjer, (2005), Giacomini and White, (2006), Patton and Timmermann, (2007), Clements and Harvey, (2010), Gneiting, (2011) and Patton, (2011) among many others.

Many important statistical functionals such as the variance, the ES, the minimum, the maximum and the mode are not elicitable, i.e. no strictly consistent loss functions exist (Gneiting, , 2011; Heinrich, , 2014; Fissler and Ziegel, , 2016). This deficiency calls for generalized approaches in many academic disciplines. We built our test procedure for the ES on such an approach, which considers multiple functionals stacked as vectors and considers joint elicitability. Fissler and Ziegel, (2016) show that the ES is jointly elicitable with the VaR by constructing strictly consistent joint loss functions for this pair, which we utilize in our encompassing approach.

In the following section, we formally introduce the concept of forecast encompassing in the classical case of one-dimensional, real-valued and elicitable functionals. Subsequently, we make use of the higher-order elicitability of the ES and generalize the encompassing approach to ES forecasts in Section 2.2.

2.1 The Encompassing Principle

Following e.g. Hendry and Richard, (1982), Mizon and Richard, (1986), Diebold, (1989) and Giacomini and Komunjer, (2005), we formally introduce the classical concept of linear forecast encompassing for one-dimensional, real-valued and elicitable functionals. We assume that two competing forecasters predict the variable of interest $Y_{t+1}$ and issue one-step ahead point forecasts \hat{\boldsymbol{f}_{t}}=\big{(}\hat{f}_{1,t},\hat{f}_{2,t}\big{)} for a given functional $\Gamma(F_{t})$.333While we focus our approach on one-step ahead forecasts, extensions to multi-step ahead forecasts are straight-forward by employing a HAC-type estimator for the asymptotic covariance. In order to conduct the forecast evaluation in an out-of-sample fashion, we divide the sample size $T$ in an in-sample part of size $m$ and an out-of-sample part of size $n$ such that $T=m+n$. The in-sample period is used to generate the forecasts $\hat{f}_{1,t}$ and $\hat{f}_{2,t}$ as described in the beginning of Section 2, while the out-of-sample period is used for the evaluation of the forecasts. This procedure poses little restrictions on how to generate the forecasts and allows for parametric, semiparametric or nonparametric techniques and for nested and non-nested forecasting procedures (Giacomini and Komunjer, , 2005).

Let \rho\big{(}Y_{t+1},\hat{f}_{t}\big{)} be a strictly consistent loss function for $\Gamma(\cdot)$. Then, we say that forecast $\hat{f}_{1,t}$ encompasses $\hat{f}_{2,t}$ at time $t$, if

[TABLE]

for all \big{(}\theta_{1},\theta_{2}\big{)}\in\Theta\subseteq{\mathbb{R}^{2}}. Equation (2.1) implies that, in terms of the loss induced by $\rho$, the forecast $\hat{f}_{1,t}$ is at least as good as any (linear) combination of $\hat{f}_{1,t}$ and $\hat{f}_{2,t}$. Hence, forecast $\hat{f}_{2,t}$ does not add any information on $Y_{t+1}$ which is not already incorporated in $\hat{f}_{1,t}$. We define \big{(}\theta^{\ast}_{1},\theta^{\ast}_{2}\big{)} as the optimal combination parameters which minimize the expected loss,

[TABLE]

By definition, it holds that \mathbb{E}\left[\rho\big{(}Y_{t+1},\theta_{1}\hat{f}_{1,t}+\theta_{2}\hat{f}_{2,t}\big{)}\right]\geq\mathbb{E}\left[\rho\big{(}Y_{t+1},\theta^{\ast}_{1}\hat{f}_{1,t}+\theta^{\ast}_{2}\hat{f}_{2,t}\big{)}\right] for all \big{(}\theta_{1},\theta_{2}\big{)}\in\Theta. In particular, this implies that

[TABLE]

Combining (2.1) and (2.3) yields the following definition of forecast encompassing.

Definition 2.1 (Linear Forecast Encompassing for Elicitable Functionals).

We say that forecast $\hat{f}_{1,t}$ encompasses $\hat{f}_{2,t}$ at time $t$ with respect to the loss function $\rho$ if and only if

[TABLE]

which is equivalent to \big{(}\theta^{\ast}_{1},\theta^{\ast}_{2}\big{)}=\big{(}1,0\big{)}.

Tests for forecast encompassing are carried out through the following steps. First, we regress the realizations $Y_{t+1}$ onto the forecasts $\hat{f}_{1,t}$ and $\hat{f}_{2,t}$ using an appropriate regression technique for the functional under consideration in order to obtain the estimated combination (or encompassing) parameters $\hat{\theta}_{n}$ and their asymptotic distribution. Then, we test whether these parameters equal one and zero respectively.

As discussed e.g. in Clements and Harvey, (2009) and Clements and Harvey, (2010), there exist several different testing specifications available for the encompassing principle, which differ in terms of the admissible specifications of the linear (or nonlinear) forecast combination formula. We generalize and unify these approaches by introducing a general link or combination function,

[TABLE]

which maps the forecasts and the respective parameters onto a linear or nonlinear forecast combination and where $\mathfrak{F}$ denotes the random space of the issued forecasts. For this, the function $g$ and the parameter space $\Theta$ have to be chosen such that there exists a $\theta_{0}\in\Theta$, such that $g(\hat{\boldsymbol{f}_{t}},\theta_{0})=\hat{f}_{1,t}$ almost surely, which enables testing whether $\hat{f}_{1,t}$ alone captures the full information provided by any forecast combination through testing the parametric restriction $\theta^{\ast}=\theta_{0}$.

Definition 2.2 (General Forecast Encompassing for Elicitable Functionals).

We say that forecast $\hat{f}_{1,t}$ encompasses $\hat{f}_{2,t}$ at time $t$ with respect to the loss function $\rho$ and with respect to the link function $g$ if and only if

[TABLE]

which is equivalent to $\theta^{\ast}=\theta_{0}$.

This general definition unifies the following existing specifications of forecast encompassing, but also allows for more general linear and nonlinear specifications, see e.g. Ericsson, (1993), Clements and Harvey, (2009) and Clements and Harvey, (2010).

Example 2.3.

Prominent examples for linear and nonlinear forecast encompassing are the following link functions and associated null hypotheses,

1. (1)

$g(\hat{\boldsymbol{f}_{t}},\theta)=\theta_{1}+\theta_{2}\hat{f}_{1,t}+\theta_{3}\hat{f}_{2,t}$ and $\mathbb{H}_{0}:(\theta_{2}^{\ast},\theta_{3}^{\ast})=(1,0)$ or $\mathbb{H}_{0}:(\theta_{1}^{\ast},\theta_{2}^{\ast},\theta_{3}^{\ast})=(0,1,0)$, 2. (2)

$g(\hat{\boldsymbol{f}_{t}},\theta)=\theta_{1}+\theta_{2}\hat{f}_{1,t}+(1-\theta_{2})\hat{f}_{2,t}$ and $\mathbb{H}_{0}:\theta_{2}^{\ast}=1$ or $\mathbb{H}_{0}:(\theta_{1}^{\ast},\theta_{2}^{\ast})=(0,1)$, 3. (3)

$g(\hat{\boldsymbol{f}_{t}},\theta)=\theta_{1}+\hat{f}_{1,t}+\theta_{2}\hat{f}_{2,t}$ and $\mathbb{H}_{0}:\theta_{2}^{\ast}=0$ or $\mathbb{H}_{0}:(\theta_{1}^{\ast},\theta_{2}^{\ast})=(0,0)$, 4. (4)

$g(\hat{\boldsymbol{f}_{t}},\theta)=\theta_{1}\hat{f}_{1,t}+\theta_{2}\hat{f}_{2,t}$ and $\mathbb{H}_{0}:(\theta_{1}^{\ast},\theta_{2}^{\ast})=(1,0)$, 5. (5)

$g(\hat{\boldsymbol{f}_{t}},\theta)=\theta_{1}\hat{f}_{1,t}+(1-\theta_{1})\hat{f}_{2,t}$ and $\mathbb{H}_{0}:\theta_{1}^{\ast}=1$, 6. (6)

$g(\hat{\boldsymbol{f}_{t}},\theta)=\hat{f}_{1,t}+\theta_{1}\hat{f}_{2,t}$ and $\mathbb{H}_{0}:\theta_{1}^{\ast}=0$, 7. (7)

g(\hat{\boldsymbol{f}_{t}},\theta)=\theta_{1}\pm\exp\big{(}\theta_{2}\log(\pm\hat{f}_{1,t})+\theta_{3}\log(\pm\hat{f}_{2,t})\big{)} and $\mathbb{H}_{0}:(\theta_{2}^{\ast},\theta_{3}^{\ast})=(1,0)$.

2.2 Forecast Encompassing for the Expected Shortfall

In this section, we consider encompassing tests for the ES. For absolutely continuous distributions $F_{t}$, the ES is formally defined as

[TABLE]

where $Q_{t,\alpha}(Y_{t+1})$ denotes the conditional $\alpha$-quantile of $Y_{t+1}$ given $\mathcal{F}_{t}$. As discussed in the previous section, the main ingredient of forecast encompassing tests is the specification of the underlying loss function, which has to be associated with the risk measures we consider forecasts for. As such loss functions do not exist for the ES stand-alone, we utilize a strictly consistent joint loss function for the pair consisting of the ES and the VaR, given by Fissler and Ziegel, (2016) as

[TABLE]

where the arguments $Y$, $q_{\alpha}$ and $e_{\alpha}$ denote the return realization, the quantile and the ES respectively. As this loss function exhibits the desirable property of having loss differences which are homogeneous of order zero, it is often denoted as the FZ0-loss function, see e.g. Patton et al., (2019). While there exist infinitely many strictly consistent loss functions for the pair VaR and ES, the recent literature seems to agree upon this choice: Dimitriadis and Bayer, (2019) find that it exhibits a stable numerical performance in M-estimation and empirically yields relatively efficient parameter estimates. Nolde and Ziegel, (2017) discuss the desirable property of homogeneity of these loss functions and Patton et al., (2019), Bayer and Dimitriadis, (2020) and Taylor, (2019) use this loss function to estimate dynamic ES models.

Following the specification of a link function in (2.5), we introduce the quantile- and ES-specific link functions

[TABLE]

where $\mathfrak{Q}$ and $\mathfrak{E}$ denote the random spaces of the VaR and ES forecasts, $\Theta^{\beta}\subseteq\mathbb{R}^{k_{\beta}}$ and $\Theta^{\eta}\subseteq\mathbb{R}^{k_{\eta}}$ such that $\Theta=\Theta^{\beta}\times\Theta^{\eta}$ and $k_{\beta}+k_{\eta}=k\in\mathbb{N}$. We assume that the functions $g^{q}$, $g^{e}$ and the parameter space $\Theta$ are chosen such that there exist values $\beta_{0}\in\Theta^{\beta}$ and $\eta_{0}\in\Theta^{\eta}$, such that $g^{q}(\hat{\boldsymbol{q}}_{t},\beta_{0})=\hat{q}_{1,t}$ and $g^{e}(\hat{\boldsymbol{e}}_{t},\eta_{0})=\hat{e}_{1,t}$ almost surely.

In the following, we introduce the concept of joint forecast encompassing for the pair consisting of the VaR and the ES. Analogously to (2.2), we define the optimal combination parameters for the VaR and ES as

[TABLE]

Definition 2.4 (Joint VaR and ES Forecast Encompassing).

Let \big{(}\hat{q}_{1,t},\hat{e}_{1,t}\big{)} and \big{(}\hat{q}_{2,t},\hat{e}_{2,t}\big{)} denote pair-wise competing forecasts for the pair consisting of the conditional quantile and ES of $F_{t}$. We say that \big{(}\hat{q}_{1,t},\hat{e}_{1,t}\big{)} encompasses \big{(}\hat{q}_{2,t},\hat{e}_{2,t}\big{)} at time $t$ with respect to the link functions $g^{q}$ and $g^{e}$ if and only if

[TABLE]

where the loss function $\rho$ is given in (2.8). This holds if and only if \big{(}\beta^{\ast},\eta^{\ast}\big{)}=\big{(}\beta_{0},\eta_{0}\big{)}.

We test whether the sequence of joint quantile and ES forecasts \big{(}\hat{q}_{1,t},\hat{e}_{1,t}\big{)} encompasses the sequence \big{(}\hat{q}_{2,t},\hat{e}_{2,t}\big{)} for all $t=m,\dots,T-1$ by estimating the parameters of the following semiparametric regression,

[TABLE]

where $Q_{\alpha}(u_{t+1}^{q}|\mathcal{F}_{t})=0$ and $ES_{\alpha}(u_{t+1}^{e}|\mathcal{F}_{t})=0$ almost surely for all $t=m,\dots,T-1$ by using the M-estimation technique introduced in Patton et al., (2019) and Dimitriadis and Bayer, (2019). We then test for \big{(}\beta^{\ast},\eta^{\ast}\big{)}=\big{(}\beta_{0},\eta_{0}\big{)} using a Wald type test statistic.

Definition 2.4 develops a joint encompassing test for the VaR and ES, which is reasonable given the joint elicitability property of the VaR and ES. However, a further objective of this paper is to construct encompassing tests for the ES stand-alone, which we do in the following.

Definition 2.5 (Auxiliary ES Forecast Encompassing).

Let \big{(}\hat{q}_{1,t},\hat{e}_{1,t}\big{)} and \big{(}\hat{q}_{2,t},\hat{e}_{2,t}\big{)} denote competing forecasts for the pair consisting of the conditional quantile and ES of $F_{t}$. We say that $\hat{e}_{1,t}$ auxiliarily encompasses $\hat{e}_{2,t}$ at time $t$ with respect to the link functions $g^{q}$ and $g^{e}$ if and only if

[TABLE]

that is, if and only if $\eta^{\ast}=\eta_{0}$.444It is important to notice that the optimal combination parameter $\beta^{\ast}$ on the left-hand side of (2.14) is given by (2.11) and not in the sense of an optimal combination parameter of a restricted model.

This parameter restriction is tested using a Wald type test statistic based on the estimates of the regression setup given in (2.13). As we do not test the quantile specific parameters $\beta^{\ast}$, we do not impose that the underlying quantile forecast also encompasses its competitor under this null hypothesis. Hence, even though this test is based on the joint regression, it only tests encompassing of the ES forecasts. We call this test auxiliary ES encompassing test as it still depends on the auxiliary quantile forecasts which are used for the estimation of the optimal combination parameters.

Given that both, the VaR and ES forecasts are available, application of either the joint or auxiliary test is the most plausible approach given their joint elicitability. However, even though the emphasis of the auxiliary encompassing test is on the ES, it still requires quantile forecasts for the implementation of the parameter estimation. This can be problematic for two reasons. First, the quantile forecasts are still used in the estimation procedure and thus have an indirect effect on the parameter estimates of the ES specific parameters. E.g., the previous tests are not applicable for ES forecasts which are based on the same VaR forecasts, as this implies perfect collinearity of the quantile regressors. Second, the auxiliary test is only applicable in the setup where the person applying the test has access to the quantile forecasts. In the current implementation of the regulatory framework of the Basel Committee (Basel Committee, , 2016, 2017), the banks are only obligated to report their ES forecasts (at probability level $2.5\%$), but not the corresponding VaR forecasts. Thus, the accompanying VaR forecasts, which the ES forecasts are internally based on, are in general not available to the regulator who has to decide on an adequate risk management of the financial institution at hand.

In order to account for these scenarios, we further introduce the strict ES encompassing test, which only requires ES forecasts in the following. For this, we slightly modify the definition of (2.11) by replacing $g^{q}(\hat{\boldsymbol{q}}_{t},\beta)$ through $g^{q}(\hat{\boldsymbol{e}}_{t},\beta)$,555Note that the parameters denoted $\theta^{\ast}$ in (2.11) and in (2.15) can generally differ.

[TABLE]

Definition 2.6 (Strict ES Forecast Encompassing).

Let $\hat{e}_{1,t}$ and $\hat{e}_{2,t}$ denote competing ES forecasts of the underlying predictive distribution $F_{t}$. We say that $\hat{e}_{1,t}$ strictly encompasses $\hat{e}_{2,t}$ at time $t$ with respect to the link functions $g^{q}$ and $g^{e}$ if and only if

[TABLE]

that is, if and only if $\eta^{\ast}=\eta_{0}$.

We test whether $\hat{e}_{1,t}$ strictly encompasses $\hat{e}_{2,t}$ for all $t=m,\dots,T-1$ by setting up the slightly transformed regression

[TABLE]

where $Q_{\alpha}(u_{t+1}^{q}|\mathcal{F}_{t})=0$ and $ES_{\alpha}(u_{t+1}^{e}|\mathcal{F}_{t})=0$ almost surely for all $t=m,\dots,T-1$. The crucial difference between this test and the joint and auxiliary encompassing tests is that instead of using the quantile forecasts $\hat{\boldsymbol{q}}_{t}$ in the quantile link function $g^{q}$, we use the ES forecasts $\hat{\boldsymbol{e}}_{t}$ for both, the quantile and ES link functions $g^{q}$ and $g^{e}$. We argue that this can be seen as a best feasible solution due to the lack of loss functions for the ES stand-alone together with the necessity of developing forecast evaluation methods for the ES stand-alone due to the current setup of the Basel III regulatory framework (Basel Committee, , 2016, 2017).

The underlying idea of this test is mainly motivated by pure scale models, i.e. $Y_{t+1}=\sigma_{t}u_{t+1}$, $u_{t+1}\sim F(0,1)$, which is still the most frequently used class of models for risk management with the GARCH and stochastic volatility models as prime examples. For this model class, the VaR and ES forecasts are perfectly colinear, $\hat{e}_{t}=\frac{\xi_{\alpha}}{z_{\alpha}}\hat{q}_{t}$, where $z_{\alpha}$ and $\xi_{\alpha}$ are the $\alpha$-quantile and $\alpha$-ES of the distribution $F(0,1)$. Hence, the quantile model $g^{q}(\hat{\boldsymbol{e}}_{t},\beta)=g^{q}(\hat{\boldsymbol{q}}_{t}\xi_{\alpha}/z_{\alpha},\beta)=g^{q}(\hat{\boldsymbol{q}}_{t},\tilde{\beta})$ is correctly specified, but with transformed quantile parameters $\tilde{\beta}$.666For the prominent case of linear encompassing link formulas $g^{q}(\cdot)$, it holds that $\tilde{\beta}=\beta z_{\alpha}/\xi_{\alpha}$. As we only test on the ES-specific parameters $\eta$ as described in Definition 2.6, our test is invariant to this (often linear) transformation of the parameter $\beta$ and thus, it is correctly specified for pure scale models.

In the general case, the quantile equation can possibly be misspecified. Thus, we provide asymptotic theory under general model misspecification for the M-estimator in the following section. The potential model misspecification might bias the pseudo-true parameters and challenge the interpretability of the test decision, but we argue that this effect is negligible for this setup. First, the misspecification is only slight in the sense that daily financial return data is approximated well by pure scale processes. Second, the misspecification is indirect in the sense that while the quantile parameters are potentially misspecified, we only test the ES parameters, which are influenced by the misspecification only indirectly through the joint estimation. Furthermore, we illustrate that the performance of our strict ES encompassing test is not negatively influenced by more general data generating processes in the simulation study in Section 3 by considering GAS models with time-varying higher moments of Creal et al., (2013) and the dynamic ES models of Patton et al., (2019) and Taylor, (2019).

Tests for equal (superior) predictive ability in the sense of Diebold and Mariano, (1995), Clark and McCracken, (2001), Giacomini and White, (2006), West, (2006) and the model confidence set approach of Hansen et al., (2011) can be seen as a general alternative to encompassing tests. As these tests are directly based on the average loss difference, they can only test the predictive ability of the VaR and ES jointly. In contrast, encompassing tests are based on the regression coefficients of the semiparametric quantile and ES models and hence, only indirectly on the respective loss function. This fundamental difference allows for stand-alone encompassing tests for ES forecasts, which constitutes a great advantage for ES encompassing tests.

Strictly speaking, strict consistency of loss functions only implies that the optimal forecast exhibits the smallest possible loss in expectation. In reality however, competing forecasts are often misspecified due to estimation error or misspecified forecasting models. Patton, (2019) shows that then, the ranking induced by the loss functions can be sensitive towards the choice of (strictly consistent) loss functions or even misleading. Holzmann and Eulert, (2014) show that for competing forecasts which are based on nested information sets and which are correctly specified given their underlying (but usually incomplete) information set (auto-calibrated), applying any strictly consistent loss function results in a correct ranking of the forecasts. In our case of testing forecast encompassing, we indeed build on nested information sets as it obviously holds that \sigma\big{\{}\hat{f}_{1,t},\hat{f}_{2,t}\big{\}}\supseteq\sigma\big{\{}\hat{f}_{1,t}\big{\}}. Thus, by further assuming that the issued forecasts are auto-calibrated given the forecaster’s information set, we can conclude that the ranking implied by (2.1) is indeed the correct one and invariant towards the choice of strictly consistent loss functions.

2.3 Asymptotic Theory under Model Misspecification

In the following, we use the short notation $g^{e}_{t}(\eta)=g^{e}(\hat{\boldsymbol{e}}_{t},\eta)$ and $g^{q}_{t}(\beta)=g^{q}(\hat{\boldsymbol{q}}_{t},\beta)$ (or $g^{q}_{t}(\beta)=g^{q}(\hat{\boldsymbol{e}}_{t},\beta)$ in the case of the strict test). We define the M-estimator as

[TABLE]

and the pseudo-true parameter as777 The pseudo-true parameter can generally depend on the issued loss function, i.e. in this case on the zero homogeneous choice in (2.8).

[TABLE]

When the link (regression) functions $g^{q}(\cdot)$ and $g^{e}(\cdot)$ are correctly specified, we get that the pseudo-true parameter $\theta^{\ast}_{n}$ equals the classical true regression parameter and it is independent of the sample size $n$. We further define the corresponding identification functions, which are almost surely the derivative of the loss function $\rho$ with respect to $\theta$,

[TABLE]

We restrict our attention to processes which satisfy the following conditions.

Assumption 2.7.

We assume that

1. (a)

the process $Z_{t}$ is strong mixing of size $-r/(r-2)$ for some $r>2$, 2. (b)

the parameter space $\Theta=\Theta^{\beta}\times\Theta^{\eta}\subseteq\mathbb{R}^{k}$ is compact and non-empty, 3. (c)

the pseudo-true parameter $\theta_{n}^{\ast}$ defined in (2.19) is in the interior of $\Theta$ and is the unique minimizer of the objective function $Q_{n}^{0}(\theta)$ and the sequence \mathbb{E}_{t}\big{[}\psi\big{(}Y_{t+1},g^{q}_{t}(\beta),g^{e}_{t}(\eta)\big{)}\big{]}, defined in (2.20) is uncorrelated, 4. (d)

the distribution of $Y_{t+1}$ given $\mathcal{F}_{t}$, denoted by $F_{t}$ is absolutely continuous with continuous and strictly positive density $h_{t}$, which is bounded from above almost surely on the whole support of $F_{t}$ and Lipschitz continuous, 5. (e)

for all $\theta$ in a neighborhood of $\theta_{n}^{\ast}$, it holds that $\left|\frac{1}{g^{e}_{t}(\eta)}\right|\leq K<\infty$ for some constant $K>0$, 6. (f)

the link functions $g_{t}^{q}(\beta)$ and $g_{t}^{e}(\eta)$ are $\mathcal{F}_{t}$-measurable, twice continuously differentiable in $\theta=(\beta,\eta)$ on $\operatorname{int}(\Theta)$ almost surely and if \mathbb{P}\big{(}g_{t}^{q}(\beta_{1})=g_{t}^{q}(\beta_{2})\cap g_{t}^{e}(\eta_{1})=g_{t}^{e}(\eta_{2})\big{)}=1, then $\theta_{1}=\theta_{2}$, 7. (g)

the matrices $\Lambda_{n}$ and $\Sigma_{n}$, defined in Proposition 2.9 are positive definite with a determinant bounded away from zero for all $n$ sufficiently large, 8. (h)

it holds that $g_{t}^{q}(\beta)\leq Q$, $\nabla g_{t}^{q}(\beta)\leq Q_{1}$, $H_{t}^{q}(\beta)\leq Q_{2}$, $\nabla H_{t}^{q}(\beta)\leq Q_{3}$, and $g_{t}^{e}(\eta)\leq E$, $\nabla g_{t}^{e}(\eta)\leq E_{1}$, $H_{t}^{e}(\eta)\leq E_{2}$, $\nabla H_{t}^{e}(\eta)\leq E_{3}$, for all $\theta$ in a neighborhood of $\theta_{n}^{\ast}$, where the random variables $Q,E,Q_{1},E_{1},Q_{2},E_{2},Q_{3},E_{3}$ are all $\mathcal{F}_{t}$-measurable and for some $r>2$ (from condition (a)), the following moments are bounded (i) $\mathbb{E}[Q_{1}^{r+1}]$, (ii) $\mathbb{E}[E_{1}^{r+1}]$, (iii) $\mathbb{E}[Q_{2}^{(r+1)/2}]$, (iv) $\mathbb{E}[E_{2}^{(r+1)/2}]$, (v) $\mathbb{E}[E_{1}Q_{2}]$, (vi) $\mathbb{E}[Q_{1}Q_{2}]$, (vii) $\mathbb{E}[Q_{1}E_{2}]$, (viii) $\mathbb{E}[Q_{1}^{2}E_{1}]$, (ix) $\mathbb{E}[EE_{1}^{3}]$, (x) $\mathbb{E}[EE_{3}]$, (xi) $\mathbb{E}[EE_{1}E_{2}]$, (xii) $\mathbb{E}[QE_{1}E_{2}]$, (xiii) $\mathbb{E}[QE_{1}^{3}]$, (xiv) $\mathbb{E}[Q_{1}Q^{r}E_{1}^{r}]$, (xv) $\mathbb{E}[E_{1}^{r-1}E_{2}|Y_{t}]^{r}]$, (xvi) $\mathbb{E}[E_{1}^{r+1}|Y_{t}]^{r}]$, (xvii) $\mathbb{E}[Y_{t}^{2r}]$, 9. (i)

for any $n$, the term $\sup_{\beta\in\Theta^{\beta}}\sum_{t=m}^{T-1}\mathds{1}_{\{Y_{t+1}=g^{q}_{t}(\beta)\}}$ is almost surely bounded from above.

The following propositions show consistency and asymptotic normality of the M-estimator under potential model misspecification.

Proposition 2.8.

Given the conditions in Assumption 2.7, it holds that $\hat{\theta}_{n}-\theta_{n}^{\ast}\stackrel{{\scriptstyle\mathbb{P}}}{{\longrightarrow}}0$.

Proposition 2.9.

Given the conditions in Assumption 2.7, it holds that

[TABLE]

with $\Omega_{n}(\theta^{\ast}_{n})=\Lambda_{n}^{-1}(\theta^{\ast}_{n})\,\Sigma_{n}(\theta^{\ast}_{n})\,\Lambda_{n}^{-1}(\theta^{\ast}_{n})$, where $\Lambda_{n}(\theta^{\ast}_{n})=\begin{pmatrix}\Lambda_{n,qq}(\theta^{\ast}_{n})&\Lambda_{n,qe}(\theta^{\ast}_{n})\\ \Lambda_{n,eq}(\theta^{\ast}_{n})&\Lambda_{n,ee}(\theta^{\ast}_{n})\end{pmatrix}$, and \Sigma_{n}(\theta^{\ast}_{n})=\frac{1}{n}\sum_{t=m}^{T-1}\mathbb{E}\left[\psi\big{(}Y_{t+1},g^{q}_{t}(\beta^{\ast}_{n}),g^{e}_{t}(\eta^{\ast}_{n})\big{)}\cdot\psi\big{(}Y_{t+1},g^{q}_{t}(\beta^{\ast}_{n}),g^{e}_{t}(\eta^{\ast}_{n})\big{)}^{\top}\right]. Furthermore, the components of $\Lambda_{n}(\theta^{\ast}_{n})$ are given by

[TABLE]

where $H^{q}_{t}(\beta)$ and $H^{e}_{t}(\eta)$ are the Hessian matrices of $g_{t}^{q}(\beta)$ and $g_{t}^{e}(\eta)$ respectively.

The two preceding propositions extend the asymptotic theory of Patton et al., (2019) to the case of possibly misspecified models, and the misspecification theory for linear models of Bayer and Dimitriadis, (2020) to nonlinear models. The proofs in Appendix A combine, extend and go along the lines of the ideas of Engle and Manganelli, (2004) and Patton et al., (2019). The conditions closely resemble the regularity conditions of Patton et al., (2019). As we further allow for model misspecification, we impose the unique minimization condition (c) and slightly strengthen the moment conditions (h). In the baseline case of linear encompassing link functions $g^{q}$ and $g^{e}$, the required moment conditions simplify to those given in Bayer and Dimitriadis, (2020).

For the estimation of the asymptotic covariance matrix $\widehat{\Omega}_{n}$ under possible model misspecification, we follow the approach of Dimitriadis and Bayer, (2019) and Bayer and Dimitriadis, (2020). We deal with the three nuisance quantities in $\widehat{\Omega}_{n}$ as follows. In order to estimate the density quantile function $h_{t}(g_{t}^{q}(\beta^{\ast}_{n}))$, we follow the nid-estimator of Hendricks and Koenker, (1992). As the degree of misspecification in the investigated financial time series is small (Bayer and Dimitriadis, , 2020), we approximate $F_{t}(g_{t}^{q}(\beta^{\ast}_{n}))\approx\alpha$. For the conditional truncated variance, \operatorname{Var}_{t}\big{(}g^{q}_{t}(\beta^{\ast}_{n})-Y_{t+1}\big{|}Y_{t+1}\leq g^{q}_{t}(\beta^{\ast}_{n})\big{)}, we employ the scl-sp estimator of Dimitriadis and Bayer, (2019).

We now consider the asymptotic distributions of our three ES encompassing tests proposed in Section 2.2 under the null hypotheses and for general link functions, where we test certain $s$-dimensional ($s\in\mathbb{N},s\leq k$) sub-vectors of $\theta$. For this, let $R\in\mathbb{R}^{k\times s}$ be a selection matrix whose columns consist of $k$-dimensional Cartesian unit (column) vectors $e_{j}\in\mathbb{R}^{k}$, which are zero apart from a one in dimension $j$. E.g., when $g_{t}^{q}(\beta)$ and $g_{t}^{e}(\eta)$ equal the linear link functions with intercept, given in the first point of Example 2.3, $\theta=(\beta_{1},\beta_{2},\beta_{3},\eta_{1},\eta_{2},\eta_{3})$. Then, for the strict and auxiliary ES encompassing tests, $R=(e_{5},e_{6})$ and for the joint test $R=(e_{2},e_{3},e_{5},e_{6})$. These choices pick the respective parameters from $\theta$. Then, we define the respective test statistics by

[TABLE]

Theorem 2.10 (ES Encompassing Tests).

Given the conditions of Assumption 2.7 and given that $\widehat{\Omega}_{n}-\Omega_{n}\stackrel{{\scriptstyle\mathbb{P}}}{{\to}}0$, under the respective null hypotheses given in Definition 2.4 - 2.6, it holds that

[TABLE]

For linear link functions, this theorem implies that the limiting $\chi^{2}$ distribution of the joint test has four degrees of freedom, while the one of the strict and auxiliary tests has two degrees of freedom.

An important application of these ES encompassing tests is in the context of selecting the best-performing forecast, i.e. selecting at time $T$ a superior forecasting method for the future. This is particularly relevant as the ES is recently introduced into the Basel regulations without having proper forecast selection procedures at hand. Following Giacomini and Komunjer, (2005), we propose the following decision rule. We test the two encompassing hypotheses $\mathbb{H}_{0}^{(1)}$: $\hat{e}_{1,t}$ encompasses $\hat{e}_{2,t}$ and $\mathbb{H}_{0}^{(2)}$: $\hat{e}_{2,t}$ encompasses $\hat{e}_{1,t}$ for $t=m,\dots,T-1$. Then, there are four possible scenarios: (1) if neither $\mathbb{H}_{0}^{(1)}$ nor $\mathbb{H}_{0}^{(2)}$ are rejected, the test is not helpful for forecast selection. (2) If $\mathbb{H}_{0}^{(1)}$ is rejected while $\mathbb{H}_{0}^{(2)}$ is not rejected, we can conclude that forecast $\hat{e}_{2,t}$ does add information to forecast $\hat{e}_{1,t}$, while we cannot conclude the reverse. Thus, we decide to use the forecasting method of $\hat{e}_{2,t}$. (3) If $\mathbb{H}_{0}^{(2)}$ is rejected while $\mathbb{H}_{0}^{(1)}$ is not rejected, the same logic applies inversely and we use the forecasting method of $\hat{e}_{1,t}$. (4) If both, $\mathbb{H}_{0}^{(1)}$and $\mathbb{H}_{0}^{(2)}$are rejected, the test delivers statistical evidence that both forecasts contain exclusive information and that a forecast combination outperforms the stand-alone forecasts. Consequently, we use a combined forecast $\hat{e}_{c,t}=g^{e}(\hat{\boldsymbol{e}}_{t},\hat{\eta}_{n})$ where the estimated combination parameters $\hat{\eta}_{n}$ are obtained from the M-estimator proposed here.

Testing forecast encompassing conditional on some information set $\tilde{\mathcal{G}}_{t}=\sigma\{\boldsymbol{W}_{t}\}$ based on some $\mathcal{F}_{t}$-measurable vector of instruments $\boldsymbol{W}_{t}$ in the sense of Giacomini and Komunjer, (2005) can be facilitated through estimating the regression parameters through (overidentified) GMM-estimation instead of M-estimation. However, for the strict ES test, this approach requires asymptotic theory under model misspecification for the overidentified GMM estimator based on nonsmooth objective functions. While such theory is available for smooth moment conditions (see e.g. Hall and Inoue, (2003) and Hansen and Lee, (2019)), its generalization to nonsmooth objective functions is not straight-forward and thus, we leave conditional ES encompassing tests based on misspecified GMM-estimation for future research. The moment conditions of our unconditional approach can be interpreted as conditional encompassing with respect to the instruments $\nabla g^{q}_{t}(\beta)$ and $\nabla g^{e}_{t}(\eta)$. In the classical baseline case of linear forecast encompassing, these instruments simplify to $\hat{\boldsymbol{q}}_{t}$ and $\hat{\boldsymbol{e}}_{t}$ and thus, our approach tests conditional encompassing with respect to the information set $\mathcal{G}_{t}=\sigma\{1,\hat{\boldsymbol{q}}_{t},\hat{\boldsymbol{e}}_{t}\}\subseteq\mathcal{F}_{t}$, which in most cases already captures the most relevant information which is available.

3 Simulation Study

In this section, we evaluate the size and power properties of our three proposed ES encompassing tests and compare them to the VaR encompassing test of Giacomini and Komunjer, (2005). For this, we describe the simulation setup in Section 3.1 and we report and discuss the simulation results in Section 3.2. Section 3.3 considers three extensions of the simulation setup with respect to additional data generating processes (DGPs), loss and link functions.

3.1 The Simulation Setup

We employ the three encompassing tests based on the linear link functions $g^{q}(\hat{\boldsymbol{f}_{t}},\beta)=\beta_{1}+\beta_{2}\hat{f}_{1,t}+\beta_{3}\hat{f}_{2,t}$ and $g^{e}(\hat{\boldsymbol{e}}_{t},\eta)=\eta_{1}+\eta_{2}\hat{e}_{1,t}+\eta_{3}\hat{e}_{2,t}$, where $\hat{\boldsymbol{f}_{t}}=\hat{\boldsymbol{q}}_{t}$ for the joint and auxiliary tests and $\hat{\boldsymbol{f}_{t}}=\hat{\boldsymbol{e}}_{t}$ for the strict test, together with the parameter space $\Theta=\{\theta=(\beta,\eta)\in\mathbb{R}^{6}:||\theta||\leq K\}$.888 We choose the constant $K$ large enough such that the parameter estimation is not restricted in realistic settings but the parameter space $\Theta$ is indeed convex. For the respective encompassing tests, in each case we test the following two opposing hypotheses:

[TABLE]

In the following, we describe two DGPs where for the first, both forecasting models stem from classical GARCH models while the second considers two joint GAS models for the VaR and ES of Patton et al., (2019). For both model classes, we simulate data as a convex combination of two distinct models with a flexible convex combination weight $\pi\in[0,1]$. This implies that for $\pi=0$, the first model encompasses the second, while for $\pi=1$, the inverse holds. For all intermediate parameters $\pi\in(0,1)$, the data stems from a linear combination and both forecast encompassing null hypotheses should be rejected which indicates that a forecast combination method is preferred.

The GARCH DGP

The two GARCH models, which are calibrated to daily IBM returns, are given by $\tilde{Y}_{j,t+1}=\hat{\sigma}_{j,t}u_{t+1}$, for $j=1,2$, where $u_{t+1}\stackrel{{\scriptstyle iid}}{{\sim}}\mathcal{N}(0,1)$ and the two distinct volatility specifications are given by

[TABLE]

For both models, we obtain VaR and ES forecasts by $\hat{q}_{j,t}=z_{\alpha}\hat{\sigma}_{j,t}$ and $\hat{e}_{j,t}=\xi_{\alpha}\hat{\sigma}_{j,t}$, for $j=1,2$, where $z_{\alpha}$ and $\xi_{\alpha}$ are the $\alpha$-quantile and $\alpha$-ES of the standard normal distribution. Notice that the time index $t$ on $\hat{\sigma}_{j,t}$ indicates that it is a $\mathcal{F}_{t}$-measurable forecast for time $t+1$. While the first specification in (3.2) is a classical GARCH(1,1) model (Bollerslev, , 1986), the second specification in (3.3) follows the GJR-GARCH model of Glosten et al., (1993), which allows for a leverage effect. We simulate data from the convex combination of these processes, Y_{t+1}=\big{(}(1-\pi)\hat{\sigma}_{1,t}+\pi\hat{\sigma}_{2,t}\big{)}u_{t+1} for 21 equally spaced values of $\pi\in[0,1]$, where $u_{t+1}\stackrel{{\scriptstyle iid}}{{\sim}}\mathcal{N}(0,1)$.

The VaR/ES GAS DGP

In the second simulation setup, we implement the one-factor (1F) and two-factor (2F) GAS models for the VaR and ES of Patton et al., (2019). The 1F-GAS model evolves as

[TABLE]

The 2F-GAS model follows the specification

[TABLE]

where the forcing variable is given by \lambda_{t}=\big{(}\hat{q}_{2,t-1}(\alpha-\mathds{1}_{\{\tilde{Y}_{2,t}\leq\hat{q}_{2,t-1}\}}),\,\mathds{1}_{\{\tilde{Y}_{2,t}\leq\hat{q}_{2,t-1}\}}\tilde{Y}_{2,t}/\alpha-\hat{e}_{2,t-1}\big{)}^{\top}. For both models, $j=1,2$, we simulate \tilde{Y}_{j,t+1}\sim\mathcal{N}\big{(}\hat{\mu}_{j,t},\hat{\sigma}_{j,t}^{2}\big{)}, where the conditional mean and standard deviations are given by $\hat{\mu}_{j,t}=\hat{q}_{j,t}-z_{\alpha}\frac{\hat{e}_{j,t}-\hat{q}_{j,t}}{\xi_{\alpha}-z_{\alpha}}$ and $\hat{\sigma}_{j,t}=\frac{\hat{e}_{j,t}-\hat{q}_{j,t}}{\xi_{\alpha}-z_{\alpha}}$, such that $Q_{\alpha}(\tilde{Y}_{j,t+1}|\mathcal{F}_{t})=\hat{q}_{j,t}$ and $ES_{\alpha}(\tilde{Y}_{j,t+1}|\mathcal{F}_{t})=\hat{e}_{j,t}$ almost surely. The parameter values for this model are obtained from Table 8 of Patton et al., (2019) and correspond to calibrated parameters to daily S&P 500 returns.

In order to simulate returns which follow a convex combination of these two conditional distributions, we simulate Bernoulli draws $\pi_{t+1}\sim\operatorname{Bern}(\pi)$ for 21 equally spaced values of $\pi\in[0,1]$, and let $Y_{t+1}=(1-\pi_{t+1})\tilde{Y}_{1,t+1}+\pi_{t+1}\tilde{Y}_{2,t+1}$. Thus, for $\pi=0$, $Y_{t+1}$ follows the 1F-GAS model, for $\pi=1$, $Y_{t+1}$ follows the 2F-GAS model and for $\pi\in(0,1)$, $Y_{t+1}$ follows some convex combination of these two models.999 While generating returns stemming from convex combinations of GARCH-type volatility models is straight-forward by using convex combinations of the conditional volatilities, this is not as simple for the more general GAS models considered in this section. Consequently, we use this more involved approach based on Bernoulli draws in order to generate these convex model combinations.

Both models in the GARCH DGP generate data from a pure scale (volatility) process resulting in perfectly colinear VaR and ES forecasts. In contrast, the more general VaR/ES GAS models in the second DGP generate VaR and ES forecasts which are not colinear and consequently introduce misspecification in the quantile model of the strict ES encompassing test. As the utilized parameters are calibrated to daily financial returns, these models reflect a realistic degree of misspecification encountered in practical risk management.

3.2 Simulation Results

Table 3.2 reports the empirical sizes of the three different ES encompassing tests introduced in Section 2 together with the VaR encompassing test of Giacomini and Komunjer, (2005) at a 10$\%$ nominal significance level based on $2000$ Monte Carlo replications. Table S.1 and S.2 in the supplementary material present equivalent results for nominal sizes of 1$\%$ and 5$\%$. The column panel $\mathbb{H}_{0}^{(1)}$ indicates that we test whether model 1 encompasses model 2, while the panel $\mathbb{H}_{0}^{(2)}$ indicates the reverse.