Finite sample properties of the Buckland-Burnham-Augustin confidence interval centered on a model averaged estimator

TL;DR

This paper analyzes the finite sample and asymptotic properties of a confidence interval based on a model averaged estimator in linear regression, providing exact formulas and performance insights.

Contribution

It derives exact finite sample expressions for coverage and length of the confidence interval, advancing understanding of its performance in simple regression models.

Findings

Exact finite sample coverage formulas derived.

Performance comparison with asymptotic results.

Insights into the interval's length and coverage behavior.

Abstract

We consider the confidence interval centered on a frequentist model averaged estimator that was proposed by Buckland, Burnham & Augustin (1997). In the context of a simple testbed situation involving two linear regression models, we derive exact expressions for the confidence interval and then for the coverage and scaled expected length of the confidence interval. We use these measures to explore the exact finite sample performance of the Buckland-Burnham-Augustin confidence interval. We also explore the limiting asymptotic case (as the residual degrees of freedom increases) and compare our results for this case to those obtained for the asymptotic coverage of the confidence interval by Hjort & Claeskens (2003).

Equations

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+\mathbf{\epsilon },$$

$${\hat{\theta }}_1$$

$$\tilde{\theta }$$

$$w_1$$

$$\text{GIC}(k)=-2L_k+d(p+k-1),$$

$$L_k=-\frac{n}{2}\ln \left(2\pi \frac{{\text{RSS}}_k}{n}\right)-\frac{n}{2},$$

$$\text{GIC}(k)=\text{ constant }+n\ln \left(\frac{{\text{RSS}}_k}{n}\right)+d(p+k-1).$$

$${\text{RSS}}_1=\frac{{\hat{\tau }}^2}{v_{\tau }}+m{\hat{\sigma }}^2.$$

$$\text{GIC}(1)=\text{ constant }+n\ln \left(\frac{m{\hat{\sigma }}^2+{\hat{\tau }}^2/v_{\tau }}{n}\right)+dp$$

$$\text{GIC}(2)=\text{ constant }+n\ln \left(\frac{m{\hat{\sigma }}^2}{n}\right)+d(p+1).$$

$$w_{1}=\frac{1}{1+\exp\left(\displaystyle\frac{1}{2}\Big{(}\text{GIC}(1)-\text{GIC}(2)\Big{)}\right)}=\frac{1}{1+\left(1+\displaystyle\frac{\widehat{\gamma}^{2}}{m}\right)^{n/2}\,\exp\big{(}-d/2\big{)}}.$$

$$w_{1}(x)=\frac{1}{1+\left(1+\displaystyle\frac{x^{2}}{m}\right)^{n/2}\,\exp\big{(}-d/2\big{)}}=\frac{1}{1+\left(1+\displaystyle\frac{x^{2}}{m}\right)^{(m+p)/2}\,\exp\big{(}-d/2\big{)}},$$

$$\displaystyle\widetilde{\theta}=\widehat{\theta}-\rho\,v_{\theta}^{1/2}\,\big{(}\widehat{\tau}/v_{\tau}^{1/2}\big{)}\,w_{1}\left(\widehat{\gamma}\right)=\widehat{\theta}-\rho\,v_{\theta}^{1/2}\,\widehat{\sigma}\widehat{\gamma}\,w_{1}\left(\widehat{\gamma}\right)=\widehat{\theta}-\rho\,v_{\theta}^{1/2}\,\widehat{\sigma}\,k(\widehat{\gamma}).$$

$$\sum_{k=1}^{2}w_{k}\,\sqrt{\big{(}\text{variance of $\widehat{\theta}_{k}$ assuming ${\cal M}_{k}$ is true}\big{)}+\Big{(}E(\widehat{\theta}_{k})-\theta\Big{)}^{2}}.$$

$$\displaystyle\big{(}\text{variance of $\widehat{\theta}_{2}$ assuming ${\cal M}_{2}$ is true}\big{)}=\sigma^{2}v_{\theta}.$$

$$\displaystyle\big{(}\text{variance of $\widehat{\theta}_{1}$ assuming ${\cal M}_{1}$ is true}\big{)}=\sigma^{2}\,v_{\theta}\,(1-\rho^{2}).$$

$$\frac{m\widehat{\sigma}^{2}+\big{(}\widehat{\tau}^{2}/v_{\tau}\big{)}}{m+1}\,v_{\theta}\,(1-\rho^{2}).$$

$${\hat{\theta }}_1-\tilde{\theta }$$

$$\displaystyle\Big{(}\widehat{\theta}_{1}-\widetilde{\theta}\Big{)}^{2}$$

$$\displaystyle\Big{(}\widehat{\theta}_{2}-\widetilde{\theta}\Big{)}^{2}$$

$$\displaystyle w_{1}\big{(}\widehat{\gamma}\big{)}\left(\frac{m\widehat{\sigma}^{2}+\big{(}\widehat{\tau}^{2}/v_{\tau}\big{)}}{m+1}\,v_{\theta}\,(1-\rho^{2})+\big{(}1-w_{1}\big{(}\widehat{\gamma}\big{)}\big{)}^{2}\,\rho^{2}\,\widehat{\sigma}^{2}\,v_{\theta}\,\widehat{\gamma}^{2}\right)^{1/2}$$

$$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\left(1-w_{1}\big{(}\widehat{\gamma}\big{)}\right)\Big{(}\widehat{\sigma}^{2}\,v_{\theta}+w_{1}^{2}\big{(}\widehat{\gamma}\big{)}\,\rho^{2}\,\widehat{\sigma}^{2}\,v_{\theta}\,\widehat{\gamma}^{2}\Big{)}^{1/2}.$$

$$\displaystyle r(x,\rho)=w_{1}(x)\left(\frac{m+x^{2}}{m+1}(1-\rho^{2})+(1-w_{1}(x))^{2}\rho^{2}\,x^{2}\right)^{1/2}+(1-w_{1}(x))\Big{(}1+w_{1}^{2}(x)\,\rho^{2}\,x^{2}\Big{)}^{1/2}.$$

$$\displaystyle\left[\widetilde{\theta}\pm t_{m,1-\alpha/2}\,\widehat{\sigma}\,v_{\theta}^{1/2}\,r\big{(}\widehat{\gamma},\rho\big{)}\right]$$

$$\displaystyle\widehat{\gamma}=\widehat{\tau}/\big{(}\widehat{\sigma}\,v_{\tau}^{1/2}\big{)}=\{\widehat{\tau}/\big{(}\sigma\,v_{\tau}^{1/2}\big{)}\}\big{(}\sigma/\widehat{\sigma}\big{)}=\widetilde{\gamma}\big{/}W,$$

$$\displaystyle\left[{\begin{array}[]{c}\widehat{\theta}\\ \widetilde{\gamma}\end{array}}\right]\sim N\left(\left[{\begin{array}[]{c}\theta\\ \gamma\end{array}}\right],\left[{\begin{array}[]{cc}\sigma^{2}\,v_{\theta}&\rho\,\sigma\,{v_{\theta}}^{1/2}\\ \rho\,\sigma\,{v_{\theta}}^{1/2}&1\end{array}}\right]\right).$$

$$J=\left[\widehat{\theta}-\rho\,\sigma\,W\,v_{\theta}^{1/2}\,k\big{(}\widetilde{\gamma}/W\big{)}\pm t_{m,1-\alpha/2}\,\sigma\,W\,v_{\theta}^{1/2}\,r\big{(}\widetilde{\gamma}/W,\rho\big{)}\right].$$

$$\ell (\gamma ,w,\rho )$$

$$u(\gamma ,w,\rho )$$

$$CP(\gamma,\rho)=\int_{0}^{\infty}\int_{-\infty}^{\infty}\Psi\Big{(}\ell(y+\gamma,w,\rho),u(y+\gamma,w,\rho);\,\rho\,y,1-\rho^{2}\Big{)}\,\phi(y)\,dy\,f_{W}(w)\,dw,$$

$$SEL(\gamma ,\rho )$$

Full text

Keywords: coverage; model averaged confidence interval; scaled expected length

∗ Corresponding author. E-mail address: P.Kabaila@latrobe.edu.au

1 Introduction

Buckland et al., (1997) proposed a frequentist model averaged estimator of a general scalar parameter that is a weighted average of estimators obtained under different models. The model weights were constructed by exponentiating the Akaike Information Criterion (AIC), see Buckland et al., (1997, pp. 605–606). This kind of model weighting has been adopted in much of the later literature (Fletcher & Dillingham, (2011); Fletcher & Turek, (2011)). Buckland et al., (1997) further proposed a standard error for the model averaged estimator and constructed an approximate (Gaussian) confidence interval centered on the model averaged estimator with width determined by the standard error. The approach of Buckland et al., (1997) was enthusiastically adopted by Burnham & Anderson, (2002) and seems to be widely used in practice in the ecological literature.

Hjort & Claeskens, (2003, Section 4.3) and Claeskens & Hjort, (2008, Section 7.5.1) criticised the confidence interval proposed by Buckland et al., (1997). In the context of a general regression model, which includes linear regression and logistic regression as particular cases, they showed that the standard error based on formula (9) of Buckland et al., (1997) is asymptotically incorrect, so that the nominal coverage of the confidence interval is not the actual coverage. The analyses of Hjort & Claeskens, (2003) and Claeskens & Hjort, (2008) do not seem to have had much impact in applied fields. Although the conclusions are clear and hold for very general regression models, the results themselves are complicated, difficult to follow and, as large sample results, deemed not very relevant to practice.

Kabaila et al., (2016) set up a very simple testbed situation for evaluating the exact finite sample frequentist properties of model averaged confidence intervals. This testbed involves computing a confidence interval by model averaging over two nested linear regression models with unknown error variance, and then computing the coverage probability and scaled expected length properties of this confidence interval. The scaled expected length is the expected length of the model averaged confidence interval divided by the expected length of the standard confidence interval (with the same minimum coverage probability and for the same parameter) computed under the full model without any model selection. Its computation gives far more insight than the coverage alone, allowing us for example to see when good coverage is obtained at the expense of excessive length. The testbed was used by Kabaila et al., (2016) to evaluate both the model averaged profile likelihood confidence interval of Fletcher & Turek, (2011) and the model averaged tail area confidence interval of Turek & Fletcher, (2012). This testbed was also used by Kabaila et al., (2017) to further evaluate the tail area confidence interval of Turek & Fletcher, (2012). These papers showed that the tail area interval performs quite well provided that we do not put too much weight on the simpler of the two models. On the other hand, there are situations in which the model averaged profile likelihood intervals are worse than the standard confidence interval used after model selection but ignoring the model selection process.

Our aim is to analyse the exact finite sample properties of the Buckland et al., (1997) confidence interval using the standard error based on the formula (9) of Buckland et al., (1997) in the testbed situation of two nested linear regression models with unknown error variance. Specifically, we want to find the exact finite sample coverage probability and scaled expected length properties of this confidence interval. We also allow the use of different model selection criteria, such as the Bayesian Information Criterion (BIC) instead of AIC, to determine the model averaging weights so that one could explore the effect of changing the weights. However, the results of Kabaila et al., (2016), Kabaila et al., (2017) and Kabaila, (2018) suggest that BIC weights put too much weight on the simpler model, producing confidence intervals with poorer performance than the AIC weights.

We define the testbed situation and the parametrisation we use in Section 2. In Section 3, we obtain explicit expressions for the model averaged estimator and the standard error proposed for it by Buckland et al., (1997) in the testbed situation. These expressions together enable us to obtain an explicit expression for the confidence interval centered on the model averaged estimator proposed by Buckland et al., (1997) (which we denote $J$) in the testbed situation. In Section 4, we then derive expressions for the coverage probability and scaled expected length of the confidence interval $J$ in the testbed situation. We present numerical results for small residual degrees of freedom $m$ under various parameter settings in Section 5. We then consider the limiting case as $m\rightarrow\infty$ with the dimension of the regression parameter fixed in Section 6 and compare these with the large sample results of Hjort & Claeskens, (2003) in Subsection 6.5. We conclude with a brief discussion in Section 7.

2 Testbed model and parametrisation

Our testbed situation involves two nested linear regression models with unknown error variance which we call the full model ${\cal M}_{2}$, and a simpler model ${\cal M}_{1}$. The full model ${\cal M}_{2}$ is given by

[TABLE]

where $\boldsymbol{y}$ is an $n$-vector of random responses, $\boldsymbol{X}$ is a $n\times p$ matrix with known, linearly independent columns, $\boldsymbol{\beta}$ is an $p$-vector of unknown parameters and $\boldsymbol{\varepsilon}$ is an $n$-vector of random errors with a $N(\boldsymbol{0},\sigma^{2}\boldsymbol{I})$ distribution in which $\sigma^{2}$ is an unknown, positive parameter. We assume throughout that that $n$ and $p$ are given. Suppose that the parameter of interest is $\theta=\boldsymbol{a}^{\top}\boldsymbol{\beta}$, where $\boldsymbol{a}$ is a specified $p$-vector ($\boldsymbol{a}\neq 0$). To define the simpler model, we define another parameter $\tau=\boldsymbol{c}^{\top}\boldsymbol{\beta}-t$, where the vector $\boldsymbol{c}$ and the number $t$ are specified and $\boldsymbol{a}$ and $\boldsymbol{c}$ are linearly independent. The model ${\cal M}_{1}$ is ${\cal M}_{2}$ with $\tau=0$.

Let $\widehat{\boldsymbol{\beta}}=(\boldsymbol{X}^{\top}\boldsymbol{X})^{-1}\boldsymbol{X}^{\top}\boldsymbol{y}$ denote the least squares estimator of $\boldsymbol{\beta}$ and \widehat{\sigma}^{2}=\big{(}\boldsymbol{y}-\boldsymbol{X}\widehat{\boldsymbol{\beta}}\big{)}^{\top}\big{(}\boldsymbol{y}-\boldsymbol{X}\widehat{\boldsymbol{\beta}}\big{)}/m, where $m=n-p$, the usual unbiased estimator of $\sigma^{2}$. We set $\widehat{\theta}=\boldsymbol{a}^{\top}\widehat{\boldsymbol{\beta}}$ and $\widehat{\tau}=\boldsymbol{c}^{\top}\widehat{\boldsymbol{\beta}}-t$. Define the known quantities v_{\theta}=\text{var}\big{(}\widehat{\theta}\big{)}/\sigma^{2}=\boldsymbol{a}^{\top}(\boldsymbol{X}^{\top}\boldsymbol{X})^{-1}\boldsymbol{a}, v_{\tau}=\text{var}\big{(}\widehat{\tau}\big{)}/\sigma^{2}=\boldsymbol{c}^{\top}(\boldsymbol{X}^{\top}\boldsymbol{X})^{-1}\boldsymbol{c} and \rho=\text{corr}\big{(}\widehat{\theta},\widehat{\tau}\big{)}=\boldsymbol{a}^{\top}(\boldsymbol{X}^{\top}\boldsymbol{X})^{-1}\boldsymbol{c}/\{\boldsymbol{a}^{\top}(\boldsymbol{X}^{\top}\boldsymbol{X})^{-1}\boldsymbol{a}\;\boldsymbol{c}^{\top}(\boldsymbol{X}^{\top}\boldsymbol{X})^{-1}\boldsymbol{c}\}^{1/2}. Finally, let \gamma=\tau/\big{(}\sigma\,v_{\tau}^{1/2}\big{)} and \widehat{\gamma}=\widehat{\tau}/\big{(}\widehat{\sigma}\,v_{\tau}^{1/2}\big{)}. Note that $\gamma$ is a measure of the closeness of the models ${\cal M}_{1}$ and ${\cal M}_{2}$ and $\widehat{\gamma}$ is an estimator of that measure.

3 The model averaged estimator and its standard deviation

3.1 The model averaged estimator $\widetilde{\theta}$

Following Buckland et al., (1997, p.604), the model averaged estimator over the class of models $\{{\cal M}_{1},{\cal M}_{2}\}$ is $\widetilde{\theta}=w_{1}\widehat{\theta}_{1}+w_{2}\widehat{\theta}_{2}$, where $\widehat{\theta}_{1}$ and $\widehat{\theta}_{2}$ are estimators of $\theta$ under the models ${\cal M}_{1}$ and ${\cal M}_{2}$ respectively and $w_{1}$ and $w_{2}$, satisfying $w_{1}+w_{2}=1$, are the data-based weights for the models ${\cal M}_{1}$ and ${\cal M}_{2}$ respectively.

We can take $\widehat{\theta}_{2}=\widehat{\theta}=\boldsymbol{a}^{\top}\widehat{\boldsymbol{\beta}}$. From Kabaila & Giri, (2009, p 3421),

[TABLE]

so we can write

[TABLE]

Buckland et al., (1997, p.606) defined the model weights to be

[TABLE]

with $w_{2}=1-w_{1}$, where $\text{AIC}(k)$ is Akaike Information Criterion for model ${\cal M}_{k}$. In a slight generalisation, we replace the Akaike Information Criterion by the Generalized Information Criterion

[TABLE]

where $L_{k}$ is the likelihood for model ${\cal M}_{k}$ ($k=1,2$), $d=2$ for AIC and $d=\ln(n)$ for BIC. That is, our weights contain the Akaike Information Criterion weights as a special case. The maximum log-likelihood for model ${\cal M}_{k}$ is

[TABLE]

where $\text{RSS}_{k}$ denotes the residual sum of squares for model ${\cal M}_{k}\,(k=1,2)$. Thus

[TABLE]

Now $\text{RSS}_{2}=m\,\hat{\sigma}^{2}$ and, using the results stated by Graybill, (1976, p.222), it may be shown that

[TABLE]

Therefore

[TABLE]

and

[TABLE]

Hence

[TABLE]

It is convenient to define

[TABLE]

where we have written $w_{1}(x)$ as a function of $x$, $m$, $p$ and $d$. Let $k(x)=x\,w_{1}(x)$ so we can write (2) as

[TABLE]

3.2 The standard error of the model averaged estimator $\widetilde{\theta}$

We use formula (9) of Buckland et al., (1997) as the standard deviation of the model averaged estimator $\widetilde{\theta}$:

[TABLE]

While the derivation of this formula is not very clear, the terms in it are explicit and Buckland et al., (1997) also specify an estimator of (9) which we call the standard error of $\widetilde{\theta}$; the estimator of \big{(}\text{variance of \widehat{\theta}{k}$assuming${\cal M}{k} is true}\big{)} is obtained in the obvious way, assuming that the model ${\cal M}_{k}$ is the true model and \Big{(}E(\widehat{\theta}_{k})-\theta\Big{)}^{2} is estimated by \Big{(}\widehat{\theta}_{k}-\widetilde{\theta}\Big{)}^{2}.

We have

[TABLE]

It may be shown that

[TABLE]

The usual estimator of (4), assuming that ${\cal M}_{2}$ is the true model, is $\widehat{\sigma}^{2}\,v_{\theta}$. The usual estimator of (5), assuming that ${\cal M}_{1}$ is the true model, is

[TABLE]

Now for the remaining terms,

[TABLE]

and, similarly, \widehat{\theta}_{2}-\widetilde{\theta}=w_{1}\big{(}\widehat{\theta}_{2}-\widehat{\theta}_{1}\big{)}. Hence

[TABLE]

By (1), $\widehat{\theta}_{1}-\widehat{\theta}_{2}=\widehat{\theta}_{1}-\widehat{\theta}=-\rho\,v_{\theta}^{1/2}\,\widehat{\tau}/v_{\tau}^{1/2}=-\rho\,\widehat{\sigma}\,v_{\theta}^{1/2}\,\widehat{\gamma}$. Thus

[TABLE]

Hence, in the testbed situation, the estimator of (9) proposed by Buckland et al., (1997) is

[TABLE]

We write this as \widehat{\sigma}\,v_{\theta}^{1/2}\,r\big{(}\widehat{\gamma},\rho\big{)}, where

[TABLE]

3.3 The confidence interval for $\theta$

The confidence interval for $\theta$, centered on $\widetilde{\theta}$, proposed by Buckland et al., (1997) is

[TABLE]

We can write

[TABLE]

where \widetilde{\gamma}=\widehat{\tau}/\big{(}\sigma\,v_{\tau}^{1/2}\big{)} and $W=\widehat{\sigma}/\sigma$. To find convenient formulas for the coverage probability and the expected length of this confidence interval, we express all quantities of interest in terms of \big{(}\widehat{\theta},\widetilde{\gamma}\big{)} and $W$. We denote the pdf of $W$ by $f_{W}$. Note that \big{(}\widehat{\theta},\widetilde{\gamma}\big{)} and $W$ are independent and

[TABLE]

The confidence interval (6), with nominal coverage $1-\alpha$, is

[TABLE]

4 The coverage probability and scaled expected length of $J$

4.1 Coverage probability

The coverage probability of the confidence interval $J$ for $\theta$ is established in the following theorem.

Theorem 1**.**

*The coverage probability of the confidence interval $J$, with nominal coverage $1-\alpha$, is a function of $(\gamma,\rho)$ so we denote this coverage probability by $CP(\gamma,\rho)$. Let *

[TABLE]

Then

[TABLE]

where $\Psi(a,b;\mu,v)=P(a\leq Z\leq b)$ for $Z\sim N(\mu,v)$. For every given $\rho$, $CP(\gamma,\rho)$ is an even function of $\gamma$ and, for every given $\gamma$, $CP(\gamma,\rho)$ is an even function of $\rho$.

The proof of this result is given in Appendix A.1. It follows that, for given $m$ and $p$, we are able to describe the coverage probability of $J$ using only the parameters $|\rho|$ and $|\gamma|$.

4.2 Scaled expected length

The scaled expected length of the confidence interval $J$ for $\theta$ is established in the following theorem.

Theorem 2**.**

*The scaled expected length of the confidence interval $J$, with nominal coverage $1-\alpha$, is a function of $(\gamma,\rho)$ so we denote this scaled expected length by $SEL(\gamma,\rho)$. Then *

[TABLE]

For every given $\rho$, $SEL(\gamma,\rho)$ is an even function of $\gamma$ and, for every given $\gamma$, $SEL(\gamma,\rho)$ is an even function of $\rho$.

The proof of this result is given in Appendix A.2. It follows that, for given $m$ and $p$, we are able to describe the scaled expected length using only the parameters $|\rho|$ and $|\gamma|$.

5 Numerical results for small $\boldsymbol{m}$

We focus on the properties of the confidence interval $J$, with nominal coverage 0.95, computed using AIC weights ($d=2)$. We constructed a number of plots of the coverage probability and scaled expected length of $J$ against $|\gamma|$ for different values of $m\in\{1,2,3,10\}$, $p\in\{3,6,12,124\}$ and $|\rho|\in\{0.2,0.5,0.7,0.9\}$. Some explanation of how we carried out the calculations for these plots is included in the Supplementary Material. We present here a selection of these plots; additional plots are included in the Supplementary Material. Recall that in (3), we expressed the weights used in the model averaged estimator in terms of $m$ and $p$; since $m=n-p$, we can use any pair of $m$, $p$ and $n=m+p$ and we choose to use $m$ and $p$.

Generally, the plots show that coverage of $J$ approaches to the nominal level as $|\gamma|$ increases whereas the scaled expected length of $J$ does not necessarily approach $1$ (as we might hope) as $|\gamma|$ increases. The minimum coverage probability of $J$ is a decreasing continuous function of $|\rho|$. Also, the minimum coverage probability of $J$ is a decreasing continuous function of $m$. When $m=1$, the coverage probability is extremely close to the nominal coverage for any given $|\rho|$ and decreases as $m$ increases.

Consider Figures 1 and 2 for $m=1$ and for $|\rho|=0.5$ and $0.9$, respectively. The minimum coverage probability of $J$ is very close to the nominal coverage $0.95$. The scaled expected length of $J$ is substantially less than 1 when $\gamma=0$ and, although the scaled expected length of $J$ does not converge to 1 as $|\gamma|\rightarrow\infty$, the maximum value of the scaled expected length of $J$ is not too much larger than $1$. The results for different $p\in\{3,6,12,24\}$ are similar. In these cases, when $m=1$, the model averaged confidence interval $J$ has good properties. Figures 3 and 4 for $m=10$ and for $|\rho|=0.5$ and $0.9$, respectively, show that the minimum coverage probability of $J$ is much lower than the nominal coverage $0.95$ and the scaled expected length of $J$ can be much larger than $1$. The scaled expected length of $J$ has a maximum value that is an increasing function of $|\rho|$, that can be much larger than $1$ for $|\rho|$ large and $m$ not small. That is, the performance of the confidence interval $J$ deteriorates as $m$ increases.

6 The case that $\boldsymbol{p\geq 3}$ is fixed and $\boldsymbol{m=n-p\rightarrow\infty}$

Suppose that $p$ is a fixed integer, satisfying $p\geq 3$, and let $n\rightarrow\infty$, so that $m\rightarrow\infty$. We describe the limiting behaviour of the confidence interval $J$ and the limits of the coverage probability and scaled expected length, as $m\rightarrow\infty$. Fully rigorous proofs of the results are laborious and are not included in this paper; figures in the Supplementary Material confirm numerically that the stated limits hold.

Assume that $\bm{D}=\lim_{n\rightarrow\infty}\boldsymbol{X}^{\top}\boldsymbol{X}/n$ exists and is nonsingular. Recall that $v_{\theta}=\boldsymbol{a}^{\top}(\boldsymbol{X}^{\top}\boldsymbol{X})^{-1}\boldsymbol{a}$ and

[TABLE]

Although not made explicit in the notation, $v_{\theta}$ and $\rho$ are functions of $n$. Let

[TABLE]

Note that $v_{\theta}\downarrow 0$ and $\rho\rightarrow\bar{\rho}$ as $n\rightarrow\infty$.

6.1 Limiting behaviour of the confidence interval $\boldsymbol{J}$

Let

[TABLE]

and $k^{*}(x)=x\,w_{1}^{*}(x)$. Also let

[TABLE]

Finally, let

[TABLE]

This interval describes the limiting behaviour of the confidence interval $J$ in the sense described below.

For each fixed $x\in\mathbb{R}$,

[TABLE]

as $m\rightarrow\infty$. Hence, for each fixed $x\in\mathbb{R}$, $k(x)=x\,w_{1}(x)\rightarrow k^{*}(x)$ as $m\rightarrow\infty$. Finally, for each fixed $x\in\mathbb{R}$, $r(x,\rho)\rightarrow r^{*}(x,\bar{\rho})$ as $m\rightarrow\infty$.

We compare the differences between the centres and half-widths of $J$ and $J^{*}$ with $\sigma\,v_{\theta}^{1/2}$, the standard deviation of $\widehat{\theta}$. Since $t_{m,1-\alpha/2}\rightarrow z_{1-\alpha/2}$ and $W\xrightarrow{p}1$, as $m\rightarrow\infty$, the differences between the centers and half-widths of $J$ and $J^{*}$, divided by $\sigma\,v_{\theta}^{1/2}$ converge in probability to 0 as $m\rightarrow\infty$.

6.2 Limiting behaviour of the coverage probability

It follows from the proof of Theorem 1 that the coverage probability

[TABLE]

where $G=(\widehat{\theta}-\theta)/(\sigma\,v_{\theta}^{1/2})$ and

[TABLE]

Let

[TABLE]

Thus

[TABLE]

so that

[TABLE]

Since the distribution of $G$ conditional on $\widetilde{\gamma}=h$ is N\big{(}\rho(h-\gamma),1-\rho^{2}\big{)},

[TABLE]

say. Consequently, P\big{(}\theta\in J\big{)}\rightarrow CP^{*}(\gamma,\bar{\rho}) as $m\rightarrow\infty$.

6.3 Limiting behaviour of the scaled expected length

We have shown that the scaled expected length of the confidence interval $J$ is

[TABLE]

It follows from 6.1.47 of Abramowitz & Stegun, (1964, p.257) that

[TABLE]

Also, $t_{m,1-\alpha/2}\rightarrow z_{1-\alpha/2}$ and $t_{m,(1+c_{\rm min})/2}-z_{(1+c_{\rm min}^{*})/2}\rightarrow 0$, as $m\rightarrow\infty$, where $c_{\rm min}^{*}$ is $CP^{*}(\gamma,\bar{\rho})$ minimized with respect to $\gamma\geq 0$. In addition, it is plausible that

[TABLE]

as $m\rightarrow\infty$. Therefore the difference between the scaled expected length of the confidence interval $J$ and

[TABLE]

say, approaches 0 as $m\rightarrow\infty$. Consequently, the scaled expected length of $J$ converges to $SEL^{*}(\gamma,\bar{\rho})$ as $m\rightarrow\infty$.

6.4 Some numerical results for large $m$

For $\bar{\rho}=0$, the interval $J^{*}$ reduces to the standard $1-\alpha$ confidence interval for $\theta$ based on the full model, assuming that $\sigma^{2}$ is known. Consistently with this fact, for $\bar{\rho}=0$, $CP^{*}(\gamma,\bar{\rho})=1-\alpha$ and $SEL^{*}(\gamma,\bar{\rho})=1$ for all $\gamma$. As $|\bar{\rho}|$ increases, $CP^{*}(\gamma,\bar{\rho})$ and $SEL^{*}(\gamma,\bar{\rho})$ increasingly differ from these values.

We first did some calculations to explore empirically the reasonableness of the limiting results stated above. In particular, for the confidence interval $J$, with nominal coverage 0.95 and constructed using AIC weights ($d=2$), we constructed figures showing the coverage probability and the scaled expected length in the case $p=4$ and $|\rho|=0.2,0.5,0.7,0.9$ at different values of $m=10,50,200,\infty$. These figures (included in the Supplementary Material) show the convergence of the coverage probability and scaled expected length to the limits stated above as $m$ increases. For all the values of $|\rho|$ we considered, the exact results for $m=200$ are very close to the limiting results, indicating that the asymptotic results are useful at this value. These figures also show that, other than for small $|\rho|$, the performance of the confidence interval $J$ deteriorates in terms of both coverage probability and scaled expected length as $m\rightarrow\infty$.

For the confidence interval $J$, with nominal coverage 0.95 and constructed using AIC weights ($d=2$), we present the coverage probability and the scaled expected length (Figure 5) in the limiting case $m\rightarrow\infty$ with $p$ fixed ($p\geq 3$) at different values of $|\bar{\rho}|=0.2,0.5,0.7,0.9$. These figures quantify how the performance of $J$ deteriorates with increasing $|\bar{\rho}|$. As expected, for small $|\bar{\rho}|$, the asymptotic coverage is the same as the nominal coverage and the scaled expected length is 1. However, with large $|\bar{\rho}|$, the minimum asymptotic coverage of the confidence interval, with nominal coverage 0.95, can be as low as 0.83, even though the scaled expected length is well above $1$ for all $|\gamma|$.

6.5 Comparison with asymptotic results of Hjort & Claeskens, (2003)

Hjort & Claeskens, (2003, p. 886) consider two nested general regression models: the full model (which they call the extended model) and the simpler model (which they call the narrow model), where the simpler model is obtained from the full model by setting a scalar parameter to a given value. In the solid line curves in their Figure 2, Hjort & Claeskens, (2003, p. 886) present the limiting coverage of the Buckland et al., (1997) confidence interval using the standard error based on formula (9) of Buckland et al., (1997) in the following context. They consider two situations corresponding to two values of a parameter that they denote by $\rho$ and which, to avoid confusion, we will denote by $\rho_{HC}$. In the caption of Figure 2, Hjort & Claeskens, (2003) define

[TABLE]

with $K$ defined at the start of their Section 3.1, $\omega$ defined in their equation (3.2) and $\tau_{0}^{2}$ just below their equation (4.3). In the Supplementary Material it is shown that $\rho_{HC}$, expressed in our notation, is -\bar{\rho}\big{/}(1-\bar{\rho}^{2})^{1/2}. Note that when $\rho_{HC}=2/3$ and $\rho_{HC}=1$, our $|\bar{\rho}|$ is equal to $2/\sqrt{13}$ and $1/\sqrt{2}$, respectively. Figure 6 shows the coverage probability of $J$ using our computations in the same situations as those considered by Hjort & Claeskens, (2003); we observe that this our Figure 6 is identical to the solid line curves of Figure 2 of Hjort & Claeskens, (2003).

7 Discussion

In the context of a simple testbed situation involving two linear regression models, we have derived exact expressions for the coverage probability and scaled expected length of the confidence interval centered on a frequentist model averaged estimator proposed by Buckland et al., (1997). Using these expressions to explore the exact finite sample performance of the Buckland-Burnham-Augustin confidence interval, we showed that the confidence interval with residual degrees of freedom $m=1$ has good coverage and scaled expected properties and that these deteriorate as $m$ increases, being already quite poor for $m=10$. We also explored the limiting asymptotic case (as $m\rightarrow\infty$) and showed that the minimum limiting coverage can be much lower than the nominal value even when the maximum scaled expected length is much larger than one, throughout the parameter space. Differences in generality and notation mean that it is not obvious how our limiting coverage results relate to those of Hjort & Claeskens, (2003) (who did not include any results on expected length). We were able to compare our results to those obtained for the asymptotic coverage of the confidence interval by Hjort & Claeskens, (2003) and show that they are the same. Our results enhance the coverage result obtained by Hjort & Claeskens, (2003) by providing exact results in the more limited testbed situation for any sample size for both coverage and scaled expected length. All the results taken together show that the Buckland-Burnham-Augustin confidence interval cannot be generally recommended.

Acknowledgment

This work was supported by an Australian Government Research Training Program Scholarship.

Appendix A Appendix

A.1 Proof of Theorem 1

(a) The coverage probability of the confidence interval $J$ is

[TABLE]

where $G=(\widehat{\theta}-\theta)/(\sigma\,v_{\theta}^{1/2})$. Note that

[TABLE]

so the distribution of $G$ conditional on $\widetilde{\gamma}=h$ is N\big{(}\rho(h-\gamma),1-\rho^{2}\big{)}. Recall that

[TABLE]

Therefore the coverage probability is

[TABLE]

where $\Psi(a,b;\mu,v)=P(a\leq Z\leq b)$ for $Z\sim N(\mu,v)$. Now, by changing the variable of integration in the inner integral to $y=h-\gamma$, we obtain (14). ∎

(b) We use the following lemmas.

Lemma 1**.**

$\Psi(a,b;\mu,v)=\Psi(-b,-a;-\mu,v)$, where $\Psi(a,b;\mu,v)=P(a\leq Z\leq b)$ for $Z\sim N(\mu,v)$.

Lemma 1 is same as the Lemma 2 of Kabaila & Wijethunga, (2019). The following lemma has some similarities to Lemma 3 of Kabaila & Wijethunga, (2019).

Lemma 2**.**

1. (i)

$-u(-h,w,\rho)=\ell(h,w,\rho)$** 2. (ii)

$-\ell(-h,w,\rho)=u(h,w,\rho)$** 3. (iii)

$\ell(h,w,-\rho)=-u(h,w,\rho)$** 4. (iv)

$u(h,w,-\rho)=-\ell(h,w,\rho)$**

Proof of Lemma 1: For $Z\sim N(\mu,v)$,

[TABLE]

∎

Proof of Lemma 2: Remember that

[TABLE]

where

[TABLE]

Obviously, $w_{1}(x)$ is an even function and $k(x)$ is an odd function. Note that $r(x,\rho)$ is an even function of $x$, for given $\rho$, and an even function of $\rho$, for given $x$.

1. (i)

Since $k$ is an odd function and $r(x,\rho)$ is an even function of $x$,

[TABLE] 2. (ii)

Since $k$ is an odd function and $r(x,\rho)$ is an even function of $x$,

[TABLE] 3. (iii)

Since $r(x,\rho)$ is an even function of $\rho$,

[TABLE] 4. (iv)

Since $r(x,\rho)$ is an even function of $\rho$,

[TABLE]

∎

From (17)

[TABLE]

Consider

[TABLE]

from Lemma 1 and since $\phi$ is an even function. Now by changing the variable of integration in the inner integral to $y=-h$,

[TABLE]

Therefore, $CP(\gamma,\rho)$ is an even function of $\gamma$, for given $\rho$.

Now consider

[TABLE]

Therefore, $CP(\gamma,\rho)$ is an even function of $\rho$, for given $\gamma$. ∎

A.2 Proof of Theorem 2

(a) The scaled expected length is

[TABLE]

Observe that

[TABLE]

Let $c_{\rm min}$ be the minimum coverage probability of the confidence interval $J$. Then the standard confidence interval with coverage $c_{\rm min}$ is \big{[}\widehat{\theta}\pm t_{m,(1+c_{\rm min})/2}\,\widehat{\sigma}\,v_{\theta}^{1/2}\big{]}. Thus

[TABLE]

Therefore the scaled expected length

[TABLE]

Note that W\sim\big{(}Q/m\big{)}^{1/2} where $Q\sim\chi^{2}_{m}$. Therefore

[TABLE]

Thus

[TABLE]

Now, by changing the variable of integration in the inner integral to $y=h-\gamma$, we obtain (15).

∎

(b) From (18)

[TABLE]

We known that $r(x,\rho)$ is an even function of $x$, for given $\rho$, and an even function of $\rho$, for given $x$. Consider the inner integral

[TABLE]

which depends on $\gamma$ and $\rho$. If we can show that $SEL_{1}(\gamma,\rho)$ is an even function of $\gamma$, for given $\rho$, and an even function of $\rho$, for given $\gamma$, then we can say that $SEL(\gamma,\rho)$ is also an even function of $\gamma$, for given $\rho$, and an even function of $\rho$, for given $\gamma$. Now consider

[TABLE]

Therefore $SEL_{1}(\gamma,\rho)$ is an even function of $\gamma$, for given $\rho$. Thus $SEL(\gamma,\rho)$ is an even function of $\gamma$, for given $\rho$. Consider

[TABLE]

Therefore $SEL_{1}(\gamma,\rho)$ is an even function of $\rho$, for given $\gamma$. Thus $SEL(\gamma,\rho)$ is an even function of $\rho$, for given $\gamma$. ∎