Convergence of Oja's online principal component flow

TL;DR

This paper establishes the global convergence, stable manifolds, and exponential convergence rates of Oja's online PCA flow by analyzing it as a Landau-Lifshitz-Gilbert equation on the Stiefel manifold, introducing a new technique based on initial data rank.

Contribution

It provides the first complete analysis of convergence properties for an online PCA flow, including explicit stable manifolds and convergence rates.

Findings

Proved global convergence of the Oja's PCA flow.

Derived explicit stable manifolds based on initial data rank.

Established exponential convergence depending on spectrum gap.

Abstract

Online principal component analysis (PCA) has been an efficient tool in practice to reduce dimension. However, convergence properties of the corresponding ODE are still unknown, including global convergence, stable manifolds, and convergence rate. In this paper, we focus on the stochastic gradient ascent (SGA) method proposed by Oja. By regarding the corresponding ODE as a Landau-Lifshitz-Gilbert (LLG) equation on the Stiefel manifold, we proved global convergence of the ODE. Moreover, we developed a new technique to determine stable manifolds. This technique analyzes the rank of the initial datum. Using this technique, we derived the explicit expression of the stable manifolds. As a consequence, exponential convergence to stable equilibrium points was also proved. The success of this new technique should be attributed to the semi-decoupling property of the SGA method: iteration of previous components does not depend on that of later ones. As far as we know, our result is the first complete one on the convergence of an online PCA flow, providing global convergence, explicit characterization of stable manifolds, and closed formula of exponential convergence depending on the spectrum gap.