Biologically inspired learning in a layered neural net

TL;DR

This paper explores biologically inspired learning rules in layered neural networks, demonstrating that Hebbian and Anti-Hebbian learning emerge naturally under certain assumptions, with network performance optimized by dilution of connections.

Contribution

It introduces a biologically motivated framework showing that only Hebbian and Anti-Hebbian learning are possible, and identifies optimal network configurations with diluted connections.

Findings

Hebbian learning occurs when output is correct

Anti-Hebbian learning occurs when output is wrong

Diluted networks perform best with realistic thresholds

Abstract

A feed-forward neural net with adaptable synaptic weights and fixed, zero or non-zero threshold potentials is studied, in the presence of a global feedback signal that can only have two values, depending on whether the output of the network in reaction to its input is right or wrong. It is found, on the basis of four biologically motivated assumptions, that only two forms of learning are possible, Hebbian and Anti-Hebbian learning. Hebbian learning should take place when the output is right, while there should be Anti-Hebbian learning when the output is wrong. For the Anti-Hebbian part of the learning rule a particular choice is made, which guarantees an adequate average neuronal activity without the need of introducing, by hand, control mechanisms like extremal dynamics. A network with realistic, i.e., non-zero threshold potentials is shown to perform its task of realizing the desired input-output relations best if it is sufficiently diluted, i.e. if only a relatively low fraction of all possible synaptic connections is realized.