A significance test for forward stepwise model selection

TL;DR

This paper develops an exact significance test for forward stepwise model selection, accommodating grouped variables and complex models, thus improving the validity of inference after model selection.

Contribution

It introduces a new statistic with an exact null distribution for forward stepwise selection, handling grouped variables and complex models like hierarchical interactions.

Findings

The $T\chi$ statistic has a truncated $\chi$ distribution under the null.

The method is computationally efficient and addresses invalid test statistics post-model selection.

Applicable to hierarchical interactions and adaptive additive models.

Abstract

We apply the methods developed by Lockhart et al. (2013) and Taylor et al. (2013) on significance tests for penalized regression to forward stepwise model selection. A general framework for selection procedures described by quadratic inequalities includes a variant of forward stepwise with grouped variables, allowing us to handle categorical variables and factor models. We provide an algorithm to compute a new statistic with an exact null distribution conditional on the outcome of the model selection procedure. This new statistic, which we denote $T\chi$, has a truncated $\chi$ distribution under the global null. We apply this test in forward stepwise iteratively on the residual after each step. The resulting method has the computational strengths of stepwise selection and addresses the problem of invalid test statistics due to model selection. We illustrate the flexibility of this method by applying it to several specialized applications of forward stepwise including a hierarchical interactions model and a recently described additive model that adaptively chooses between linear and nonlinear effects for each variable.