Spectral Guarantees for Adversarial Streaming PCA

TL;DR

This paper establishes spectral ratio thresholds for effective adversarial streaming PCA, introduces a new analysis of Oja's algorithm, and demonstrates fundamental space complexity limits for the problem.

Contribution

It provides the first spectral-tailored analysis of Oja's algorithm for adversarial streams and identifies necessary spectral ratio bounds for streaming PCA.

Findings

Mergeable summaries require spectral ratio R = Ω(√d).

A variant of Oja's algorithm achieves low error with R = O(log n log d).

No algorithms with sub-quadratic space can succeed at R = O(1).

Abstract

In streaming PCA, we see a stream of vectors $x_1, \dotsc, x_n \in \mathbb{R}^d$ and want to estimate the top eigenvector of their covariance matrix. This is easier if the spectral ratio $R = \lambda_1 / \lambda_2$ is large. We ask: how large does $R$ need to be to solve streaming PCA in $\widetilde{O}(d)$ space? Existing algorithms require $R = \widetilde{\Omega}(d)$. We show: (1) For all mergeable summaries, $R = \widetilde{\Omega}(\sqrt{d})$ is necessary. (2) In the insertion-only model, a variant of Oja's algorithm gets $o(1)$ error for $R = O(\log n \log d)$. (3) No algorithm with $o(d^2)$ space gets $o(1)$ error for $R = O(1)$. Our analysis is the first application of Oja's algorithm to adversarial streams. It is also the first algorithm for adversarial streaming PCA that is designed for a spectral, rather than Frobenius, bound on the tail; and the bound it needs is exponentially better than is possible by adapting a Frobenius guarantee.