Jointly Sparse Global SIMPLS Regression

TL;DR

This paper introduces a global variable selection method for PLS regression that encourages shared variable selection across components, improving interpretability and reducing overfitting in high-dimensional settings.

Contribution

It formulates PLS regression with joint sparsity as a variational optimization problem using a novel global SIMPLS criterion and proposes an augmented Lagrangian method for efficient solution.

Findings

Achieves comparable or better predictive performance with fewer variables.

Demonstrates effectiveness on high-dimensional datasets.

Provides a new optimization framework for sparse PLS regression.

Abstract

Partial least squares (PLS) regression combines dimensionality reduction and prediction using a latent variable model. Since partial least squares regression (PLS-R) does not require matrix inversion or diagonalization, it can be applied to problems with large numbers of variables. As predictor dimension increases, variable selection becomes essential to avoid over-fitting, to provide more accurate predictors and to yield more interpretable parameters. We propose a global variable selection approach that penalizes the total number of variables across all PLS components. Put another way, the proposed global penalty encourages the selected variables to be shared among the PLS components. We formulate PLS-R with joint sparsity as a variational optimization problem with objective function equal to a novel global SIMPLS criterion plus a mixed norm sparsity penalty on the weight matrix. The mixed norm sparsity penalty is the $\ell_1$ norm of the $\ell_2$ norm on the weights corresponding to the same variable used over all the PLS components. A novel augmented Lagrangian method is proposed to solve the optimization problem and soft thresholding for sparsity occurs naturally as part of the iterative solution. Experiments show that the modified PLS-R attains better or as good performance with many fewer selected predictor variables.