On the explanatory power of principal components

TL;DR

This paper investigates the probabilistic behavior of vectors relative to orthogonal bases in high-dimensional spaces and discusses implications for Principal Components Analysis in regression and learning contexts.

Contribution

It provides a probabilistic analysis of vector proximity in high-dimensional orthogonal bases and explores its implications for PCA's explanatory power.

Findings

Probability of a vector being closer to all basis vectors than other vectors approaches 1/2 as dimension increases.

Distribution of the vector's proximity converges to a normal distribution on [-1,1] with increasing dimension.

Results have significant implications for PCA in regression and learning settings.

Abstract

We show that if we have an orthogonal base ($u_1,\ldots,u_p$) in a $p$-dimensional vector space, and select $p+1$ vectors $v_1,\ldots, v_p$ and $w$ such that the vectors traverse the origin, then the probability of $w$ being to closer to all the vectors in the base than to $v_1,\ldots, v_p$ is at least 1/2 and converges as $p$ increases to infinity to a normal distribution on the interval [-1,1]; i.e., $\Phi(1)-\Phi(-1)\approx0.6826$. This result has relevant consequences for Principal Components Analysis in the context of regression and other learning settings, if we take the orthogonal base as the direction of the principal components.