Diffusion approximations of Oja's online principal component analysis

TL;DR

This paper analyzes the convergence of Oja's online PCA algorithm by approximating it with stochastic differential equations on the Stiefel manifold, demonstrating its long-term stability and performance.

Contribution

It introduces a diffusion approximation framework for Oja's algorithm, establishing weak convergence and long-time stability on the Stiefel manifold.

Findings

SDEs approximate the discrete algorithm effectively

Weak convergence of the stochastic process is proven

Long-time convergence ensures algorithm stability

Abstract

Oja's algorithm of principal component analysis (PCA) has been one of the methods utilized in practice to reduce dimension. In this paper, we focus on the convergence property of the discrete algorithm. To realize that, we view the algorithm as a stochastic process on the parameter space and semi-group. We approximate it by SDEs, and prove large time convergence of the SDEs to ensure its performance. This process is completed in three steps. First, the discrete algorithm can be viewed as a semigroup: $S^k\varphi=\mathbb{E}[\varphi(\mathbf W(k))]$. Second, we construct stochastic differential equations (SDEs) on the Stiefel manifold, i.e. the diffusion approximation, to approximate the semigroup. By proving the weak convergence, we verify that the algorithm is 'close to' the SDEs. Finally, we use the reversibility of the SDEs to prove long-time convergence.