Global stability of a Hebbian/anti-Hebbian network for principal subspace learning

TL;DR

This paper proves the global stability of a biologically inspired neural network model that learns principal subspaces through Hebbian and anti-Hebbian updates, clarifying its convergence behavior.

Contribution

It establishes the first proof of global stability for the nonlinear synaptic dynamics in a Hebbian/anti-Hebbian network for principal subspace learning.

Findings

Synaptic weights converge to an orthonormal invariant manifold.

The network dynamics evolve in two phases: convergence to the manifold and gradient flow on a potential.

The model's minima span the principal subspace of input data.

Abstract

Biological neural networks self-organize according to local synaptic modifications to produce stable computations. How modifications at the synaptic level give rise to such computations at the network level remains an open question. Pehlevan et al. [Neur. Comp. 27 (2015), 1461--1495] proposed a model of a self-organizing neural network with Hebbian and anti-Hebbian synaptic updates that implements an algorithm for principal subspace analysis; however, global stability of the nonlinear synaptic dynamics has not been established. Here, for the case that the feedforward and recurrent weights evolve at the same timescale, we prove global stability of the continuum limit of the synaptic dynamics and show that the dynamics evolve in two phases. In the first phase, the synaptic weights converge to an invariant manifold where the `neural filters' are orthonormal. In the second phase, the synaptic dynamics follow the gradient flow of a non-convex potential function whose minima correspond to neural filters that span the principal subspace of the input data.