On confidence intervals centered on bootstrap smoothed estimators

TL;DR

This paper evaluates confidence intervals based on bootstrap smoothed estimators, extending previous work to unknown error variance, and finds conditions where these intervals perform well.

Contribution

It derives an exact formula for the standard deviation approximation of bootstrap smoothed estimators with unknown variance and assesses their performance.

Findings

Confidence intervals can perform well under unknown variance.

Performance depends on specific circumstances and model settings.

Extension of previous known-variance results to unknown-variance case.

Abstract

Bootstrap smoothed (bagged) estimators have been proposed as an improvement on estimators found after preliminary data-based model selection. Efron, 2014, derived a widely applicable formula for a delta method approximation to the standard deviation of the bootstrap smoothed estimator. He also considered a confidence interval centered on the bootstrap smoothed estimator, with width proportional to the estimate of this standard deviation. Kabaila and Wijethunga, 2019, assessed the performance of this confidence interval in the scenario of two nested linear regression models, the full model and a simpler model, for the case of known error variance and preliminary model selection using a hypothesis test. They found that the performance of this confidence interval was not substantially better than the usual confidence interval based on the full model, with the same minimum coverage. We extend this assessment to the case of unknown error variance by deriving a computationally convenient exact formula for the ideal (i.e. in the limit as the number of bootstrap replications diverges to infinity) delta method approximation to the standard deviation of the bootstrap smoothed estimator. Our results show that, unlike the known error variance case, there are circumstances in which this confidence interval has attractive properties.

Figures

Equations

$$\mbox{$\bm{y}$}=\bm{X}\bm{\beta}+\bm{\varepsilon}$$

$${\hat{\theta }}_{\text{textsc}PMS}=\{\begin{array}{l}\hat{\theta }-\frac{v_{\theta \tau }}{v_{\tau }}\hat{\tau }\text{ if }|\hat{\gamma }|\le t_m(\tilde{\alpha })\\ \hat{\theta }\text{ otherwise}.\end{array}$$

$$\frac{1}{B}\sum_{i=1}^{B}g\big{(}\widehat{\bm{\beta}}_{i}^{*},\widehat{\sigma}_{i}^{*}\big{)}.$$

$$\int_{0}^{\infty}\Big{(}\phi(d_{m}w+\gamma)-\phi(d_{m}w-\gamma)+\gamma\big{(}\Phi(d_{m}w-\gamma)-\Phi(-d_{m}w-\gamma)\big{)}\Big{)}\,f_{W}(w)\,dw,$$

$$\tilde{\theta }=\hat{\theta }-\rho \hat{\sigma }{v}_{\theta }^{1/2}{k}_m(\hat{\gamma }).$$

$$r_{\,\rm delta}(\gamma)=\Bigg{(}\frac{\rho^{2}}{2n}\Big{(}k_{m}(\gamma)+h_{m}(\gamma)-\gamma\,q_{m}(\gamma)\Big{)}^{2}+1-2\rho^{2}q_{m}(\gamma)+\rho^{2}q_{m}^{2}(\gamma)\Bigg{)}^{1/2}.$$

$$\int_{0}^{\infty}\big{(}-d_{m}w\,\phi(d_{m}w+\gamma)-d_{m}w\,\phi(d_{m}w-\gamma)+\Phi(d_{m}w-\gamma)-\Phi(-d_{m}w-\gamma)\big{)}\,f_{W}(w)\,dw$$

$$h_{m}(\gamma)=\int_{0}^{\infty}\Big{(}(d_{m}w)^{2}\phi(d_{m}w+\gamma)-(d_{m}w)^{2}\phi(d_{m}w-\gamma)\Big{)}\,f_{W}(w)\,dw,$$

$$J_{\mathrm{delta}}$$

$$\displaystyle=\Big{[}\widetilde{\theta}-t_{m}(\alpha)\,\widehat{\sigma}\,v_{\theta}^{1/2}\,r_{\,\rm delta}(\widehat{\gamma}),\,\widetilde{\theta}+t_{m}(\alpha)\,\widehat{\sigma}\,v_{\theta}^{1/2}\,r_{\,\rm delta}(\widehat{\gamma})\Big{]},$$

$$\ell (h,w,\rho )$$

$$u(h,w,\rho )$$

$$\int_{0}^{\infty}\int_{-\infty}^{\infty}\Psi\Big{(}\ell(y+\gamma,w,\rho),u(y+\gamma,w,\rho);\rho(y),1-\rho^{2}\Big{)}\phi(y)\,dy\,f_{W}(w)\,dw,$$

$$\frac{t_m(\alpha )}{t_m(1-c_{\min })}{\left(\frac{m}{2}\right)}^{1/2}\frac{\Gamma (m/2)}{\Gamma ((m+1)/2)}{\int }_0^{\infty }{\int }_{-\infty }^{\infty }w{r}_{\mathrm{delta}}\left(\frac{y+\gamma }{w}\right)\varphi (y)dy{f}_W(w)dw.$$

$$\displaystyle\widehat{\bm{s}}=\left[\begin{array}[]{c}\bm{y}^{\top}\bm{y}\\ \widehat{\bm{\beta}}\end{array}\right],\quad\bm{\eta}=\left[\begin{array}[]{c}-1/(2\sigma^{2})\\ \bm{X}^{\top}\bm{X}\bm{\beta}/\sigma^{2}\end{array}\right]\quad\text{and}\quad\psi(\bm{\eta})=\frac{\bm{\beta}^{\top}\bm{X}^{\top}\bm{X}\bm{\beta}}{2\sigma^{2}}+\frac{n}{2}\log(\sigma^{2}).$$

$${\bf sd}_{\rm delta}=\Big{(}\big{(}\mbox{$\textrm{{cov}}$}_{*}(\bm{\eta})\big{)}^{\top}\,\big{(}V(\bm{\eta})\big{)}^{-1}\,\mbox{$\textrm{{cov}}$}_{*}(\bm{\eta})\Big{)}^{1/2},$$

$$V(\mathbf{\eta })$$

$$\displaystyle\mbox{$\textrm{{var}}$}(\bm{y}^{\top}\bm{y})$$

$$=4{\sigma }^2{\mathbf{\beta }}^{\top }{\mathbf{X}}^{\top }\mathbf{X}\mathbf{\beta }+2n{\sigma }^4.$$

$$V(\mathbf{\eta })={\sigma }^2\left[\text{vbox}\text{raise}0.0pt\text{vbox}\right]$$

$$\big{(}V(\bm{\eta})\big{)}^{-1}=\frac{1}{\sigma^{2}}\left[\vbox{\hbox{\kern-1.15198pt\raise 0.0pt\hbox{\kern 151.02101pt}\kern 0.0pt\vbox{}\kern-1.15198pt}}\right].$$

$$\displaystyle E\Big{(}\big{(}\widehat{\bm{s}}-\bm{s}\big{)}\big{(}\widehat{\theta}_{\textsc{\tiny PMS}}-\theta\big{)}\Big{)}$$

$$\displaystyle=E\Big{(}\big{(}\widehat{\bm{s}}-\bm{s}\big{)}\big{(}\widehat{\theta}-\theta\big{)}\Big{)}-\rho\,\sigma\,v_{\theta}^{1/2}\int_{0}^{\infty}\int_{-d_{m}w}^{d_{m}w}z\,E\big{(}\widehat{\bm{s}}-\bm{s}\big{|}\widetilde{\gamma}=z\big{)}\phi(z-\gamma)\,dz\,f_{W}(w)\,dw$$

$$\displaystyle E\Big{(}\big{(}\widehat{\bm{s}}-\bm{s}\big{)}\big{(}\widehat{\theta}-\theta\big{)}\Big{)}=\sigma^{2}\left[\begin{array}[]{c}2\,\theta\\ v_{\theta}\\ \rho\,v_{\theta}^{1/2}\,v_{\tau}^{1/2}\\ \bm{0}\end{array}\right]$$

$$\displaystyle E\big{(}\widehat{\bm{s}}-\bm{s}\big{|}\widetilde{\gamma}=z\big{)}$$

$$\displaystyle=\left[\begin{array}[]{c}\sigma^{2}\Big{(}2\gamma(z-\gamma)+(z-\gamma)^{2}-1\Big{)}\\ \sigma(z-\gamma)\left[\begin{array}[]{c}\rho\,v_{\theta}^{1/2}\\ v_{\tau}^{1/2}\\ \bm{0}\end{array}\right]\end{array}\right].$$

$$\displaystyle\int_{0}^{\infty}\int_{-d_{m}w}^{d_{m}w}z\,E\big{(}\widehat{\bm{s}}-\bm{s}\big{|}\widetilde{\gamma}=z\big{)}\phi(z-\gamma)dz\,f_{W}(w)\,dw$$

$$\displaystyle\qquad\qquad\qquad\qquad=\sigma\left[\begin{array}[]{c}\sigma\Big{(}\gamma\,q_{m}(\gamma)+k_{m}(\gamma)+h_{m}(\gamma)\Big{)}\\ \rho\,v_{\theta}^{1/2}q_{m}(\gamma)\\ v_{\tau}^{1/2}q_{m}(\gamma)\\ \bm{0}\end{array}\right],$$

$$\mbox{$\textrm{{cov}}$}_{*}(\bm{\eta})=\sigma^{2}\left[\begin{array}[]{c}2\theta-\rho\,\sigma v_{\theta}^{1/2}\Big{(}\gamma\,q_{m}(\gamma)+k_{m}(\gamma)+h_{m}(\gamma)\Big{)}\\ v_{\theta}\,\big{(}1-\rho^{2}\,q_{m}(\gamma)\big{)}\\ \rho\,v_{\theta}^{1/2}v_{\tau}^{1/2}\big{(}1-q_{m}(\gamma)\big{)}\\ \bm{0}\end{array}\right].$$

$$P\left(\tilde{\theta }-t_m(\alpha ){\mathbf{sd}}_{\mathrm{delta}}(\hat{\gamma },\hat{\sigma })\le \theta \le \tilde{\theta }+t_m(\alpha ){\mathbf{sd}}_{\mathrm{delta}}(\hat{\gamma },\hat{\sigma })\right)$$

$$=P\left(-t_m(\alpha )\frac{{\mathbf{sd}}_{\mathrm{delta}}(\hat{\gamma },\hat{\sigma })}{\hat{\sigma }{v}_{\theta }^{1/2}}\le \frac{\hat{\theta }-\theta }{\hat{\sigma }{v}_{\theta }^{1/2}}-\rho k_m(\hat{\gamma })\le t_m(\alpha )\frac{{\mathbf{sd}}_{\mathrm{delta}}(\hat{\gamma },\hat{\sigma })}{\hat{\sigma }{v}_{\theta }^{1/2}}\right)$$

$$=P\left(-t_m(\alpha )r_{\mathrm{delta}}(\hat{\gamma })\le \frac{G}{W}-\rho k_m(\hat{\gamma })\le t_m(\alpha )r_{\mathrm{delta}}(\hat{\gamma })\right)$$

$$\displaystyle=P\Bigg{(}-t_{m}(\alpha)\,r_{\rm delta}\left(\frac{\widetilde{\gamma}}{W}\right)\leq\frac{G}{W}-\rho\,k_{m}\left(\frac{\widetilde{\gamma}}{W}\right)\leq t_{m}(\alpha)\,r_{\rm delta}\left(\frac{\widetilde{\gamma}}{W}\right)\Bigg{)}$$

$$\displaystyle=\int_{0}^{\infty}\int_{-\infty}^{\infty}P\Bigg{(}-t_{m}(\alpha)\,r_{\rm delta}\left(\frac{\widetilde{\gamma}}{W}\right)\leq\frac{G}{W}-\rho\,k_{m}\left(\frac{\widetilde{\gamma}}{W}\right)\leq t_{m}(\alpha)\,r_{\rm delta}\left(\frac{\widetilde{\gamma}}{W}\right)\Bigg{|}$$

$$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\widetilde{\gamma}=h,W=w\Bigg{)}\phi(h-\gamma)\,dh\,f_{W}(w)\,dw.$$

$$\displaystyle\int_{0}^{\infty}\int_{-\infty}^{\infty}P\Bigg{(}-t_{m}(\alpha)\,r_{\rm delta}\left(\frac{h}{w}\right)\leq\frac{G}{w}-\rho\,k_{m}\left(\frac{h}{w}\right)\leq t_{m}(\alpha)\,r_{\rm delta}\left(\frac{h}{w}\right)\Bigg{|}\widetilde{\gamma}=h\Bigg{)}$$

Full text

[TABLE]

Here $q_{m}(\gamma)$ is defined to be

[TABLE]

and

[TABLE]

where, as before, $d_{m}=t_{m}(\widetilde{\alpha})$.

We expect, intuitively, that the results obtained for the case that $\sigma^{2}$ is unknown (so that it must be estimated from the data) and $m\rightarrow\infty$ should be the same as for the case that $\sigma^{2}$ is known. Suppose that $p$ is fixed and $n\rightarrow\infty$, so that $m=n-p$ also diverges to $\infty$. As expected, the ideal delta method approximation to the standard deviation of $\widetilde{\theta}$ given by Theorem 1 converges to the corresponding quantity given by Theorem 2 of Kabaila and Wijethunga (2019), which deals with the case that $\sigma^{2}$ is known.

4. Computationally convenient exact formula for the coverage probability of the confidence interval centered on the bootstrap smoothed estimator

Consider the CI for $\theta$ centered on the bootstrap smoothed estimator $\widetilde{\theta}$, with nominal coverage $1-\alpha$,

[TABLE]

which we call the ${\bf sd}_{\rm delta}$ interval. Note that when $\rho=0$, this CI is identical to the usual CI, with actual coverage $1-\alpha$, based on the full model ${\cal M}_{2}$. It may be shown that the coverage probability $P(\theta\in J_{\rm delta})$ is a function of $(\gamma,\rho)$. We therefore denote this coverage probability by ${\rm CP}_{\rm delta}(\gamma,\rho)$. The following theorem is proved in Appendix A.2.

[TABLE]

Then ${\rm CP}_{\rm delta}(\gamma,\rho)$ is given by

[TABLE]

where \Psi\big{(}\ell,u;\mu,v\big{)}=P\big{(}\ell\leq Z\leq u\big{)} for $Z\sim N(\mu,v)$.

The expression (2) suggests that, for all sufficiently large $n$, ${\rm CP}_{\rm delta}(\gamma,\rho)$ is determined by $m$, for any given $(\gamma,\rho)$. Computational results for $n=25$ (described later in this section) and $n=100$ (not described either here or in the Supporting Material) suggest that, for all $n\geq 25$, ${\rm CP}_{\rm delta}(\gamma,\rho)$ is, for practical purposes, determined by $m$, for any given $(\gamma,\rho)$. It may be shown that ${\rm CP}_{\rm delta}(\gamma,\rho)$ is (a) an even function of $\gamma$ for each $\rho$ and (b) an even function of $\rho$ for each $\gamma$. It follows that, for given $n$ and $m$, we are able to encapsulate the coverage probability of the ${\bf sd}_{\rm delta}\,{\bf interval}$, for all possible choices of design matrix, parameter of interest $\theta$ and parameter $\tau$ that specifies the simpler model, using only the parameters $|\rho|$ and $|\gamma|$.

Figure 1 is the graph of coverage probability of the confidence interval $J_{\rm delta}$ centered on the bootstrap smoothed estimator, which is based on the post-model-selection estimator obtained after a preliminary hypothesis test, with size $\widetilde{\alpha}=0.1$, of the null hypothesis that the simpler model is correct. We consider the case that the nominal coverage is 0.95, $n=25$, $m=1$ and $|\rho|=0.2,0.5,0.7$ and 0.9. All of the computations reported in this paper were carried out using programs written in R. The minimum coverage probability of this CI is a continuous decreasing function of $|\rho|$ which equals the nominal coverage when $\rho=0$. Graphs of the coverage probability of $J_{\rm delta}$ for the same values of nominal coverage, size of the preliminary hypothesis test, $n$ and $|\rho|$ are provided in the Supporting Material for $m=2,3$ and 10. Further extensive numerical investigations, not reported either here or in the Supporting Material, show that the ${\bf sd}_{\rm delta}\,{\bf interval}$ outperforms the post-model-selection CI, with the same nominal coverage and based on the same preliminary test, in terms of coverage probability.

5. Computationally convenient exact formula for the scaled expected length of the confidence interval centered on the bootstrap smoothed estimator

We define the scaled expected length of $J_{\rm delta}$, with nominal coverage $1-\alpha$, to be the expected length of $J_{\rm delta}$ divided by the expected length of the usual CI, based on the full model, with the same coverage as the minimum coverage probability of $J_{\rm delta}$. Let $c_{\rm min}$ denote this minimum coverage probability. Now let $I(c)$ denote the usual CI for $\theta$, with coverage probability $c$, based on the full model. In other words, I(c)=\Big{[}\widehat{\theta}-t_{m}(1-c)\,\widehat{\sigma}\,v_{\theta}^{1/2},\,\widehat{\theta}+t_{m}(1-c)\,\widehat{\sigma}\,v_{\theta}^{1/2}\Big{]}. It may be shown that the scaled expected length of $J_{\rm delta}$ is a function of $(\gamma,\rho)$. We therefore denote this scaled expected length by ${\rm SEL}_{\rm delta}(\gamma,\rho)$. The following theorem is proved in Appendix A.3.

[TABLE]

The expression (2) suggests that, for all sufficiently large $n$, ${\rm SEL}_{\rm delta}(\gamma,\rho)$ is determined by $m$, for any given $(\gamma,\rho)$. Computational results for $n=25$ (described later in this section) and $n=100$ (not described either here or in the Supporting Material) suggest that, for all $n\geq 25$, ${\rm SEL}_{\rm delta}(\gamma,\rho)$ is, for practical purposes, determined by $m$, for any given $(\gamma,\rho)$. It may be shown that ${\rm SEL}_{\rm delta}(\gamma,\rho)$ is (a) an even function of $\gamma$ for each $\rho$ and (b) an even function of $\rho$ for each $\gamma$. It follows that, for given $n$ and $m$, we are able to encapsulate the scaled expected length of the ${\bf sd}_{\rm delta}\,{\bf interval}$, for all possible choices of design matrix, parameter of interest $\theta$ and parameter $\tau$ that specifies the simpler model, using only the parameters $|\rho|$ and $|\gamma|$.

The bootstrap smoothed estimator is obtained by smoothing the post-model-selection estimator that results from a preliminary test of the null hypothesis that the simpler model is correct i.e. that $\gamma=0$. This post-model-selection estimator is usually motivated by a desire for good performance when the simpler model is correct. Therefore, ideally, the ${\bf sd}_{\rm delta}\,{\bf interval}$ should have a scaled expected length that is substantially less than 1 when $\gamma=0$. In addition, ideally, this confidence interval should have a scaled expected length that (a) has maximum value that is not too much larger than 1 and (b) approaches 1 as $|\gamma|$ approaches infinity.

Figure 2 is the graph of scaled expected length of the confidence interval centered on the bootstrap smoothed estimator, which is based on the post-model-selection estimator obtained after a preliminary hypothesis test, with size $\widetilde{\alpha}=0.1$, of the null hypothesis that the simpler model is correct. We consider the case that the nominal coverage is 0.95, $n=25$, $m=1$ and $|\rho|=0.2,0.5,0.7$ and 0.9. For $|\rho|=0.5,0.7$ and 0.9, the scaled expected length is substantially less than 1 when $\gamma=0$. In addition, the scaled expected length (a) has maximum value that is not too much larger than 1 and (b) approaches 1 as $|\gamma|$ approaches infinity. This shows that for $m=1$ and $|\rho|\geq 0.5$ the scaled expected length of ${\bf sd}_{\rm delta}$ interval has the desired properties. This finding is similar to that reported in Kabaila and Giri (2013) concerning the performance of the CIs constructed by Kabaila and Giri (2009) to have the desired coverage probability and these desired scaled expected length properties. Namely, the performance of this CI improves as $|\rho|$ increases and $m$ decreases.

By contrast, for the case that $\sigma^{2}$ is assumed known, examined by Kabaila and Wijethunga (2019), the scaled expected length of the CI centered on the bootstrap smoothed estimator (a) is either greater than 1 or only slightly less than 1 at $\gamma=0$ and (b) has maximum value that is an increasing function of $|\rho|$ that can be much larger than 1 for large $|\rho|$. As noted earlier, we expect that as $m$ increases (which implies that $n$ also increases), the results obtained in the present paper will approach the corresponding results obtained by Kabaila and Wijethunga (2019). Therefore we expect that as $m$ increases the ${\bf sd}_{\rm delta}$ interval will get further and further away from possessing the desired scaled expected length properties. This is confirmed by the graphs of the scaled expected length of $J_{\rm delta}$ for nominal coverage 0.95, size $\widetilde{\alpha}=0.1$ of the preliminary hypothesis test, $n=25$ and $|\rho|\in\{0.2,0.5,0.7,0.9\}$ that are provided in the Supporting Material for $m=2,3$ and 10.

6. Discussion

For the test scenario of two nested linear regression models and error variance assumed known, Kabaila and Wijethunga (2019) found that the ${\bf sd}_{\rm delta}$ interval does not perform any better in terms of expected length than the usual confidence interval, with the same minimum coverage probability and based on the full model. Intuitively, the case that the error variance is assumed to be known corresponds to the case that the error variance is unknown (so that it must be estimated) and the number of degrees of freedom $m$ for the estimation of the error variance is large.

In the present paper, we deal with the case that the error variance is unknown. We find that, for small $m$ and large magnitude of correlation between the least squares estimators of the parameter of interest and the parameter that is set to zero to specify the simpler model, the expected length of the ${\bf sd}_{\rm delta}$ interval possesses some attractive features.

Acknowledgement

This work was supported by an Australian Government Research Training Program Scholarship.