Delete or merge regressors for linear model selection

TL;DR

This paper introduces the delete or merge regressors (DMR) algorithm for linear model selection with mixed predictors, demonstrating its consistency, efficiency, and improved accuracy over Lasso-based methods through theoretical proofs and simulations.

Contribution

The paper presents a novel stepwise backward algorithm called DMR for selecting linear models with mixed predictors, including a version for generalized linear models.

Findings

DMR is consistent as predictors and sample size grow.

DMR outperforms Lasso in speed and accuracy in simulations.

A generalized linear model version of DMR is proposed.

Abstract

We consider a problem of linear model selection in the presence of both continuous and categorical predictors. Feasible models consist of subsets of numerical variables and partitions of levels of factors. A new algorithm called delete or merge regressors (DMR) is presented which is a stepwise backward procedure involving ranking the predictors according to squared t-statistics and choosing the final model minimizing BIC. In the article we prove consistency of DMR when the number of predictors tends to infinity with the sample size and describe a simulation study using a pertaining R package. The results indicate significant advantage in time complexity and selection accuracy of our algorithm over Lasso-based methods described in the literature. Moreover, a version of DMR for generalized linear models is proposed.