Adaptive Forward Stepwise Regression

TL;DR

Adaptive Forward Stepwise Regression (AFS) offers a new sparse regression approach that balances the sparsity of Forward Stepwise selection with the stability of LASSO, improving performance in both regression and classification tasks.

Contribution

The paper introduces AFS, a novel method that interpolates between FS and LASSO, providing sparser models with shrinkage and demonstrating its effectiveness through simulations and real data.

Findings

AFS produces sparser models than LASSO with comparable or better accuracy.

AFS demonstrates lower mean squared error in simulations and real data.

AFS adapts easily to classification tasks.

Abstract

This paper proposes a sparse regression method that continuously interpolates between Forward Stepwise selection (FS) and the LASSO. When tuned appropriately, our solutions are much sparser than typical LASSO fits but, unlike FS fits, benefit from the stabilizing effect of shrinkage. Our method, Adaptive Forward Stepwise Regression (AFS) addresses this need for sparser models with shrinkage. We show its connection with boosting via a soft-thresholding viewpoint and demonstrate the ease of adapting the method to classification tasks. In both simulations and real data, our method has lower mean squared error and fewer selected features across multiple settings compared to popular sparse modeling procedures.