On the Optimality of the Oja's Algorithm for Online PCA

TL;DR

This paper proves that Oja's algorithm for online PCA converges efficiently and optimally, matching the best possible approximation bounds for sub-Gaussian data distributions.

Contribution

It provides the first proof that Oja's algorithm achieves a convergence rate matching the offline PCA lower bound for sub-Gaussian distributions.

Findings

Oja's algorithm converges with high probability to the principal subspace.

The convergence rate is gap-free and global.

The rate matches the offline PCA lower bound up to a constant.

Abstract

In this paper we analyze the behavior of the Oja's algorithm for online/streaming principal component subspace estimation. It is proved that with high probability it performs an efficient, gap-free, global convergence rate to approximate an principal component subspace for any sub-Gaussian distribution. Moreover, it is the first time to show that the convergence rate, namely the upper bound of the approximation, exactly matches the lower bound of an approximation obtained by the offline/classical PCA up to a constant factor.