From statistical inference to a differential learning rule for stochastic neural networks

TL;DR

This paper introduces a biologically plausible learning rule for stochastic neural networks that enables efficient pattern storage and generative modeling without complex modifications.

Contribution

The paper proposes the delayed-correlations matching (DCM) rule, a novel synaptic plasticity rule that is biologically feasible and effective for learning in stochastic neural networks.

Findings

DCM stores many patterns as attractors in stochastic networks.

It handles correlated patterns and various architectures.

It enables one-shot learning and generative modeling with hidden units.

Abstract

Stochastic neural networks are a prototypical computational device able to build a probabilistic representation of an ensemble of external stimuli. Building on the relationship between inference and learning, we derive a synaptic plasticity rule that relies only on delayed activity correlations, and that shows a number of remarkable features. Our "delayed-correlations matching" (DCM) rule satisfies some basic requirements for biological feasibility: finite and noisy afferent signals, Dale's principle and asymmetry of synaptic connections, locality of the weight update computations. Nevertheless, the DCM rule is capable of storing a large, extensive number of patterns as attractors in a stochastic recurrent neural network, under general scenarios without requiring any modification: it can deal with correlated patterns, a broad range of architectures (with or without hidden neuronal states), one-shot learning with the palimpsest property, all the while avoiding the proliferation of spurious attractors. When hidden units are present, our learning rule can be employed to construct Boltzmann machine-like generative models, exploiting the addition of hidden neurons in feature extraction and classification tasks.