Implicitly Normalized Online PCA: A Regularized Algorithm with Exact High-Dimensional Dynamics

TL;DR

This paper introduces INO-PCA, an online PCA algorithm that allows the parameter norm to evolve dynamically, revealing new insights into high-dimensional learning dynamics and improving performance over traditional methods.

Contribution

The paper proposes a novel online PCA method that relaxes norm constraints, providing a rigorous analysis of its high-dimensional dynamics and demonstrating improved empirical results.

Findings

Parameter norm follows a coupled ODE with cosine similarity.

Identifies a phase transition in steady-state performance.

Outperforms Oja's algorithm and adapts to non-stationary environments.

Abstract

Many online learning algorithms, including classical online PCA methods, enforce explicit normalization steps that discard the evolving norm of the parameter vector. We show that this norm can in fact encode meaningful information about the underlying statistical structure of the problem, and that exploiting this information leads to improved learning behavior. Motivated by this principle, we introduce Implicitly Normalized Online PCA (INO-PCA), an online PCA algorithm that removes the unit-norm constraint and instead allows the parameter norm to evolve dynamically through a simple regularized update. We prove that in the high-dimensional limit the joint empirical distribution of the estimate and the true component converges to a deterministic measure-valued process governed by a nonlinear PDE. This analysis reveals that the parameter norm obeys a closed-form ODE coupled with the cosine similarity, forming an internal state variable that regulates learning rate, stability, and sensitivity to signal-to-noise ratio (SNR). The resulting dynamics uncover a three-way relationship between the norm, SNR, and optimal step size, and expose a sharp phase transition in steady-state performance. Both theoretically and experimentally, we show that INO-PCA consistently outperforms Oja's algorithm and adapts rapidly in non-stationary environments. Overall, our results demonstrate that relaxing norm constraints can be a principled and effective way to encode and exploit problem-relevant information in online learning algorithms.