Fluctuation-learning relationship in neural networks

TL;DR

This paper establishes a theoretical link between neural activity variability and learning speed in neural networks, using fluctuation-response principles, and confirms the relationship through simulations and predictions.

Contribution

It derives formulae connecting neural activity variance with learning speed, providing a theoretical basis for empirical observations and extending understanding of neural learning dynamics.

Findings

Learning speed is proportional to neural activity variance.

Spontaneous activity variance predicts learning efficiency.

Increased neuronal gain and memory load enhance learning speed.

Abstract

Learning in neural systems occurs through change in synaptic connectivity that is driven by neural activity. Learning performance is influenced by both neural activity and the task to be learned. Experimental studies suggest a link between learning speed and variability in neural activity before learning. However, the theoretical basis of this relationship has remained unclear. In this work, using principles from the fluctuation-response relation in statistical physics, we derive two formulae that connect neural activity with learning speed. The first formula shows that learning speed is proportional to the variance of spontaneous neural activity and the neural response to input. The second formula, for small input, indicates that speed is proportional to the variances of spontaneous activity in both target and input directions. These formulae apply to various learning tasks governed by Hebbian or generalized learning rules. Numerical simulations confirm that these formulae are valid beyond their theoretical assumptions, even in cases where synaptic connectivity undergoes large changes. Our theory predicts that learning speed increases with the gain of neuronal activation functions and the number of pre-embedded memories, as both enhance the variance of spontaneous neural fluctuations. Additionally, the formulae reveal which input/output relationships are easier to learn, aligning with experimental data. Thus, our results provide a theoretical foundation for the quantitative relationship between pre-learning neural activity fluctuations and learning speed, offering insights into a range of empirical observations.