Fast Stochastic Algorithms for SVD and PCA: Convergence Properties and Convexity

TL;DR

This paper analyzes the convergence of stochastic algorithms for PCA and SVD, introducing new theoretical results, including convergence from random starts and the impact of pre-initialization, along with insights into the problem's convexity.

Contribution

It provides a formal convergence analysis of the VR-PCA algorithm, including a block version and effects of pre-initialization, and explores the convexity properties of the optimization problem.

Findings

Convergence from random initialization is established.

Pre-initialization with a single power iteration improves runtime.

The convexity and non-convexity properties of the problem are characterized.

Abstract

We study the convergence properties of the VR-PCA algorithm introduced by \cite{shamir2015stochastic} for fast computation of leading singular vectors. We prove several new results, including a formal analysis of a block version of the algorithm, and convergence from random initialization. We also make a few observations of independent interest, such as how pre-initializing with just a single exact power iteration can significantly improve the runtime of stochastic methods, and what are the convexity and non-convexity properties of the underlying optimization problem.