An extended latent factor framework for ill-posed linear regression

TL;DR

This paper extends the latent factor model for high-dimensional linear regression, analyzing the effectiveness of dimensionality reduction techniques like PLS and PCR, and providing convergence guarantees and practical stopping rules.

Contribution

It introduces an extended framework separating relevant and irrelevant features, enabling rigorous analysis of PLS and PCR in ill-posed high-dimensional regression problems.

Findings

PCR is generally unsuitable for most applications.

High-probability convergence rates for PLS are established.

Early stopping can be guided by empirical condition numbers.

Abstract

In many applications, particularly in the natural sciences, the available high-dimensional set of features may contain variables that are not correlated with the response under consideration. Such irrelevant features can, in certain cases, hinder both the accurate estimation and meaningful interpretation of the effects of the relevant features on the response. At the same time, the relevant features may also be well-approximated within a low-dimensional linear subspace, rendering the problem ill-posed. These observations motivate an extension of the classical latent factor model for linear regression. In this extended framework, it is assumed that, up to an unknown orthogonal transformation, the feature set comprises two subsets: one relevant and one irrelevant to the response. A joint low-dimensionality is imposed solely on the relevant features and the response variable. This setting enables the analysis of arbitrary linear dimensionality reduction techniques under a random design setting. In particular, it is demonstrated why principal component regression (PCR) is generally unsuitable for most applications. The framework also allows for a comprehensive analysis of the partial least squares (PLS) algorithm under random design. High-probability convergence rates are established for the sample PLS estimator with respect to an oracle latent coefficient vector, along with the corresponding linear prediction risk. Additionally, it is shown that early stopping can be guided by the empirical condition numbers of the projected design matrix. The theoretical results are validated through numerical studies on both real and simulated datasets.