PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning

TL;DR

PCA++ introduces a uniformity-constrained contrastive PCA method that enhances robustness to background noise in high-dimensional data, outperforming standard PCA in various noisy scenarios.

Contribution

The paper proposes PCA++, a novel contrastive PCA method with a closed-form solution that regularizes against background noise, improving signal recovery in high-dimensional settings.

Findings

PCA++ outperforms standard PCA and PCA+ in noisy data scenarios.

Uniformity constraints improve robustness to structured background noise.

Theoretical analysis confirms stability and regularization benefits of PCA++.

Abstract

High-dimensional data often contain low-dimensional signals obscured by structured background noise, which limits the effectiveness of standard PCA. Motivated by contrastive learning, we address the problem of recovering shared signal subspaces from positive pairs, paired observations sharing the same signal but differing in background. Our baseline, PCA+, uses alignment-only contrastive learning and succeeds when background variation is mild, but fails under strong noise or high-dimensional regimes. To address this, we introduce PCA++, a hard uniformity-constrained contrastive PCA that enforces identity covariance on projected features. PCA++ has a closed-form solution via a generalized eigenproblem, remains stable in high dimensions, and provably regularizes against background interference. We provide exact high-dimensional asymptotics in both fixed-aspect-ratio and growing-spike regimes, showing uniformity's role in robust signal recovery. Empirically, PCA++ outperforms standard PCA and alignment-only PCA+ on simulations, corrupted-MNIST, and single-cell transcriptomics, reliably recovering condition-invariant structure. More broadly, we clarify uniformity's role in contrastive learning, showing that explicit feature dispersion defends against structured noise and enhances robustness.