On stepwise regression

TL;DR

This paper introduces a simple, model-free stepwise regression method that includes covariates based on their performance relative to random noise, avoiding assumptions of a linear model with error.

Contribution

It proposes a novel, conceptually simple, and algorithmically efficient approach to variable selection that does not rely on traditional linear model assumptions.

Findings

The method effectively distinguishes relevant covariates from noise.

It provides a straightforward criterion for covariate inclusion.

The approach is computationally simple and model-agnostic.

Abstract

Given data $y$ and $k$ covariates $x$ one problem in linear regression is to decide which in any of the covariates to include when regressing $y$ on the $x$. If $k$ is small it is possible to evaluate each subset of the $x$. If however $k$ is large then some other procedure must be use. Stepwise regression and the lasso are two such procedures but they both assume a linear model with error term. A different approach is taken here which does not assume a model. A covariate is included if it is better than random noise. This defines a procedure which is simple both conceptually and algorithmically