Stochastic Optimization of PCA with Capped MSG

TL;DR

This paper introduces a stochastic optimization approach for PCA using a new Matrix Stochastic Gradient algorithm and its practical variant, Capped MSG, with theoretical analysis and empirical validation.

Contribution

It proposes a novel stochastic approximation algorithm for PCA, called Matrix Stochastic Gradient, and a practical variant named Capped MSG, advancing stochastic PCA methods.

Findings

Theoretical convergence guarantees for MSG and Capped MSG.

Empirical results demonstrating effectiveness on PCA tasks.

Comparison showing improved performance over existing methods.

Abstract

We study PCA as a stochastic optimization problem and propose a novel stochastic approximation algorithm which we refer to as "Matrix Stochastic Gradient" (MSG), as well as a practical variant, Capped MSG. We study the method both theoretically and empirically.