Deep Learning Partial Least Squares

TL;DR

This paper introduces a nonlinear deep learning extension of partial least squares (PLS) that enhances feature selection, provides model diagnostics, and quantifies uncertainty, applicable to regression and classification tasks.

Contribution

It presents a novel deep learning framework for PLS, including new models like PLS-ReLU, PLS-Autoencoder, PLS-Trees, and PLS-GP, with Bayesian interpretation and practical applications.

Findings

Effective nonlinear PLS models demonstrated on simulated data.

Model diagnostics like scree-plot and bi-plot are integrated.

Uncertainty quantification achieved via MCMC methods.

Abstract

High dimensional data reduction techniques are provided by using partial least squares within deep learning. Our framework provides a nonlinear extension of PLS together with a disciplined approach to feature selection and architecture design in deep learning. This leads to a statistical interpretation of deep learning that is tailor made for predictive problems. We can use the tools of PLS, such as scree-plot, bi-plot to provide model diagnostics. Posterior predictive uncertainty is available using MCMC methods at the last layer. Thus we achieve the best of both worlds: scalability and fast predictive rule construction together with uncertainty quantification. Our key construct is to employ deep learning within PLS by predicting the output scores as a deep learner of the input scores. As with PLS our X-scores are constructed using SVD and applied to both regression and classification problems and are fast and scalable. Following Frank and Friedman 1993, we provide a Bayesian shrinkage interpretation of our nonlinear predictor. We introduce a variety of new partial least squares models: PLS-ReLU, PLS-Autoencoder, PLS-Trees and PLS-GP. To illustrate our methodology, we use simulated examples and the analysis of preferences of orange juice and predicting wine quality as a function of input characteristics. We also illustrate Brillinger's estimation procedure to provide the feature selection and data dimension reduction. Finally, we conclude with directions for future research.