A semi-group approach to Principal Component Analysis

TL;DR

This paper presents a novel algebraic, model-based extension of PCA that applies spectral representation to distributions without second moments, connecting PCA with autoencoders and variable selection methods.

Contribution

It introduces a semi-group approach to PCA, extending its applicability to distributions lacking second moments and providing a spectral-based approximation framework.

Findings

Extends PCA to distributions without second moments.

Links PCA with autoencoders and variable selection.

Provides insights into variable selection limitations.

Abstract

Principal Component Analysis (PCA) is a well known procedure to reduce intrinsic complexity of a dataset, essentially through simplifying the covariance structure or the correlation structure. We introduce a novel algebraic, model-based point of view and provide in particular an extension of the PCA to distributions without second moments by formulating the PCA as a best low rank approximation problem. In contrast to hitherto existing approaches, the approximation is based on a kind of spectral representation, and not on the real space. Nonetheless, the prominent role of the eigenvectors is here reduced to define the approximating surface and its maximal dimension. In this perspective, our approach is close to the original idea of Pearson (1901) and hence to autoencoders. Since variable selection in linear regression can be seen as a special case of our extension, our approach gives some insight, why the various variable selection methods, such as forward selection and best subset selection, cannot be expected to coincide. The linear regression model itself and the PCA regression appear as limit cases.