Biologically Plausible Online Principal Component Analysis Without Recurrent Neural Dynamics

TL;DR

This paper introduces a biologically plausible online PCA algorithm that eliminates the need for fixed-point iterations by modifying the similarity matching objective, enabling stable, efficient, and local learning in neural networks.

Contribution

It presents a novel PCA network that avoids fast fixed-point dynamics by encouraging near-diagonality of weights, with a rigorous stability analysis and competitive convergence in online learning.

Findings

The algorithm converges at a competitive rate in online settings.

Computational complexity per iteration is linear in degrees of freedom.

The method is stable and suitable for biologically plausible neural networks.

Abstract

Artificial neural networks that learn to perform Principal Component Analysis (PCA) and related tasks using strictly local learning rules have been previously derived based on the principle of similarity matching: similar pairs of inputs should map to similar pairs of outputs. However, the operation of these networks (and of similar networks) requires a fixed-point iteration to determine the output corresponding to a given input, which means that dynamics must operate on a faster time scale than the variation of the input. Further, during these fast dynamics such networks typically "disable" learning, updating synaptic weights only once the fixed-point iteration has been resolved. Here, we derive a network for PCA-based dimensionality reduction that avoids this fast fixed-point iteration. The key novelty of our approach is a modification of the similarity matching objective to encourage near-diagonality of a synaptic weight matrix. We then approximately invert this matrix using a Taylor series approximation, replacing the previous fast iterations. In the offline setting, our algorithm corresponds to a dynamical system, the stability of which we rigorously analyze. In the online setting (i.e., with stochastic gradients), we map our algorithm to a familiar neural network architecture and give numerical results showing that our method converges at a competitive rate. The computational complexity per iteration of our online algorithm is linear in the total degrees of freedom, which is in some sense optimal.