On some limitations of probabilistic models for dimension-reduction: Illustration in the case of probabilistic formulations of partial least squares

TL;DR

This paper critically examines probabilistic formulations of Partial Least Squares (PLS), revealing their restrictive assumptions and proposing a more flexible alternative, supported by numerical illustrations and discussions on related models.

Contribution

It identifies limitations in existing probabilistic PLS models and introduces a less restrictive probabilistic formulation for PLS-SVD.

Findings

Original probabilistic PLS-SVD models are too restrictive.

Proposed alternative model relaxes distributional constraints.

Numerical examples demonstrate limitations of the original model.

Abstract

Partial Least Squares (PLS) refer to a class of dimension-reduction techniques aiming at the identification of two sets of components with maximal covariance, to model the relationship between two sets of observed variables $x\in\mathbb{R}^p$ and $y\in\mathbb{R}^q$, with $p\geq 1, q\geq 1$. Probabilistic formulations have recently been proposed for several versions of the PLS. Focusing first on the probabilistic formulation of the PLS-SVD proposed by el Bouhaddani et al., we establish that the constraints on their model parameters are too restrictive and define particular distributions for $(x,y)$, under which components with maximal covariance (solutions of PLS-SVD) are also necessarily of respective maximal variances (solutions of principal components analyses of $x$ and $y$, respectively). We propose an alternative probabilistic formulation of PLS-SVD, no longer restricted to these particular distributions. We then present numerical illustrations of the limitation of the original model of el Bouhaddani et al. We also briefly discuss similar limitations in another latent variable model for dimension-reduction.