Approximating the Span of Principal Components via Iterative Least-Squares

TL;DR

This paper introduces an iterative least-squares method to approximate the principal component space, establishing a connection between PCA and linear least-squares regression, and demonstrating convergence to the principal space.

Contribution

It proposes a novel iterative least-squares approach that converges to the principal component space, linking PCA with linear least-squares regression.

Findings

The method converges to the principal component space.

It establishes a theoretical connection between PCA and least-squares regression.

The approach provides an alternative way to approximate principal components.

Abstract

In the course of the last century, Principal Component Analysis (PCA) have become one of the pillars of modern scientific methods. Although PCA is normally addressed as a statistical tool aiming at finding orthogonal directions on which the variance is maximized, its first introduction by Pearson at 1901 was done through defining a non-linear least-squares minimization problem of fitting a plane to scattered data points. Thus, it seems natural that PCA and linear least-squares regression are somewhat related, as they both aim at fitting planes to data points. In this paper, we present a connection between the two approaches. Specifically, we present an iterated linear least-squares approach, yielding a sequence of subspaces, which converges to the space spanned by the leading principal components (i.e., principal space).