Hierarchical selection of variables in sparse high-dimensional regression

TL;DR

This paper introduces a hierarchical randomized method for variable selection in high-dimensional sparse regression models, effectively identifying relevant variables and interactions while reducing complexity.

Contribution

It proposes a novel hierarchical randomized search procedure for variable and interaction selection in high-dimensional sparse regression, improving estimation efficiency.

Findings

Effective reduction in the number of non-zero coefficients.

Maintains similar prediction loss with fewer variables.

Applicable to models with large numbers of interacting variables.

Abstract

We study a regression model with a huge number of interacting variables. We consider a specific approximation of the regression function under two ssumptions: (i) there exists a sparse representation of the regression function in a suggested basis, (ii) there are no interactions outside of the set of the corresponding main effects. We suggest an hierarchical randomized search procedure for selection of variables and of their interactions. We show that given an initial estimator, an estimator with a similar prediction loss but with a smaller number of non-zero coordinates can be found.