Resampling Sensitivity of High-Dimensional PCA

TL;DR

This paper investigates the stability of PCA under data resampling in high-dimensional settings, revealing a sharp threshold at which PCA's principal components become either stable or completely change due to small data perturbations.

Contribution

It establishes the precise threshold for PCA sensitivity to data resampling in high dimensions, linking stability to the number of resampled entries.

Findings

For large resampling ($k o ext{large}$), principal components become orthogonal.

For small resampling ($k o 0$), principal components remain colinear.

PCA's stability sharply transitions at $k oughly n^{5/3}$ resampled entries.

Abstract

The study of stability and sensitivity of statistical methods or algorithms with respect to their data is an important problem in machine learning and statistics. The performance of the algorithm under resampling of the data is a fundamental way to measure its stability and is closely related to generalization or privacy of the algorithm. In this paper, we study the resampling sensitivity for the principal component analysis (PCA). Given an $ n \times p $ random matrix $ \mathbf{X} $, let $ \mathbf{X}^{[k]} $ be the matrix obtained from $ \mathbf{X} $ by resampling $ k $ randomly chosen entries of $ \mathbf{X} $. Let $ \mathbf{v} $ and $ \mathbf{v}^{[k]} $ denote the principal components of $ \mathbf{X} $ and $ \mathbf{X}^{[k]} $. In the proportional growth regime $ p/n \to \xi \in (0,1] $, we establish the sharp threshold for the sensitivity/stability transition of PCA. When $ k \gg n^{5/3} $, the principal components $ \mathbf{v} $ and $ \mathbf{v}^{[k]} $ are asymptotically orthogonal. On the other hand, when $ k \ll n^{5/3} $, the principal components $ \mathbf{v} $ and $ \mathbf{v}^{[k]} $ are asymptotically colinear. In words, we show that PCA is sensitive to the input data in the sense that resampling even a negligible portion of the input may completely change the output.