The coverage probability of confidence intervals in one-way analysis of covariance after two F tests

TL;DR

This paper examines how a two-stage model selection procedure in one-way analysis of covariance affects the coverage probability of confidence intervals for a parameter, revealing potential inadequacies in the method.

Contribution

It provides a general methodology to assess the impact of two-stage model selection on confidence interval coverage in ANCOVA models.

Findings

The confidence interval can be completely inadequate under the two-stage procedure.

The methodology allows for evaluating coverage probability in complex model selection scenarios.

Numerical example demonstrates the potential pitfalls of the proposed procedure.

Abstract

Consider a one-way analysis of covariance model. Suppose that the parameter of interest theta is a specified linear contrast of the expected responses, for a given value of the covariate. Also suppose that the inference of interest is a 1-alpha confidence interval for theta. The following two-stage procedure has been proposed to determine the form of the model. In Stage 1, we carry out an F test of the null hypothesis that the slopes are all zero against the alternative hypothesis that they are not all zero. If this null hypothesis is accepted then we assume that the slopes are all zero; otherwise we proceed to Stage 2. In Stage 2, we carry out an F test of the null hypothesis that the slopes are all equal against the alternative hypothesis that they are not all equal. If this null hypothesis is accepted then we assume that the slopes are all equal; otherwise this assumption is not made. We present a general methodology for the examination of the effect of this two-stage model selection procedure on the coverage probability of a subsequently-constructed confidence interval for theta, with nominal coverage 1-alpha. This methodology is applied to a numerical example for which it is shown that this confidence interval is completely inadequate.