Derivation of Learning Rules for Coupled Principal Component Analysis in a Lagrange-Newton Framework

TL;DR

This paper introduces a Lagrange-Newton framework for deriving stable, convergent learning rules for principal component analysis, enabling simultaneous estimation of eigenvectors and eigenvalues through coupled differential equations.

Contribution

It presents a novel Lagrange-Newton approach to derive coupled PCA learning rules with guaranteed convergence and stability properties.

Findings

Framework guarantees equal convergence speed from all directions.

Derived coupled PCA rules estimate eigenvectors and eigenvalues simultaneously.

Feasibility demonstrated for two PCA learning rules.

Abstract

We describe a Lagrange-Newton framework for the derivation of learning rules with desirable convergence properties and apply it to the case of principal component analysis (PCA). In this framework, a Newton descent is applied to an extended variable vector which also includes Lagrange multipliers introduced with constraints. The Newton descent guarantees equal convergence speed from all directions, but is also required to produce stable fixed points in the system with the extended state vector. The framework produces "coupled" PCA learning rules which simultaneously estimate an eigenvector and the corresponding eigenvalue in cross-coupled differential equations. We demonstrate the feasibility of this approach for two PCA learning rules, one for the estimation of the principal, the other for the estimate of an arbitrary eigenvector-eigenvalue pair (eigenpair).