Regression-aware decompositions

TL;DR

This paper introduces regression-aware matrix decompositions that incorporate auxiliary information from a design matrix A into classical dimensionality reduction methods like PCA and ID, enabling supervised insights into the structure of B.

Contribution

It presents a novel framework for regression-aware decompositions that integrate supervision into PCA and ID, bridging regression models with classical unsupervised techniques.

Findings

Regression-aware ID and PCA provide supervised insights into matrix B.

These decompositions reveal structure in B relevant to regression with A.

They interpret as a form of canonical correlation analysis.

Abstract

Linear least-squares regression with a "design" matrix A approximates a given matrix B via minimization of the spectral- or Frobenius-norm discrepancy ||AX-B|| over every conformingly sized matrix X. Another popular approximation is low-rank approximation via principal component analysis (PCA) -- which is essentially singular value decomposition (SVD) -- or interpolative decomposition (ID). Classically, PCA/SVD and ID operate solely with the matrix B being approximated, not supervised by any auxiliary matrix A. However, linear least-squares regression models can inform the ID, yielding regression-aware ID. As a bonus, this provides an interpretation as regression-aware PCA for a kind of canonical correlation analysis between A and B. The regression-aware decompositions effectively enable supervision to inform classical dimensionality reduction, which classically has been totally unsupervised. The regression-aware decompositions reveal the structure inherent in B that is relevant to regression against A.