Slowly evolving geometry in recurrent neural networks I: extreme dilution regime

TL;DR

This paper analyzes a neural network model with extremely diluted, slowly evolving connectivity, revealing how phase behavior and memory stability are affected by temperature and connectivity dynamics.

Contribution

It introduces an exactly solvable model of neural networks with evolving connectivity, highlighting the impact of slow connectivity changes on phase diagrams and memory retrieval.

Findings

Retrieval phase volume diverges as connectivity temperature decreases.

Fraction of misaligned spins decreases with lowering connectivity temperature.

Memory stability is preserved despite the presence of locally stable non-condensed states.

Abstract

We study extremely diluted spin models of neural networks in which the connectivity evolves in time, although adiabatically slowly compared to the neurons, according to stochastic equations which on average aim to reduce frustration. The (fast) neurons and (slow) connectivity variables equilibrate separately, but at different temperatures. Our model is exactly solvable in equilibrium. We obtain phase diagrams upon making the condensed ansatz (i.e. recall of one pattern). These show that, as the connectivity temperature is lowered, the volume of the retrieval phase diverges and the fraction of mis-aligned spins is reduced. Still one always retains a region in the retrieval phase where recall states other than the one corresponding to the `condensed' pattern are locally stable, so the associative memory character of our model is preserved.