About Learning in Recurrent Bistable Gradient Networks

TL;DR

This paper examines the learning dynamics of Recurrent Bistable Gradient Networks, highlighting limitations of Hebbian training and proposing Contrastive Divergence as a promising alternative, tested on MNIST digit recognition.

Contribution

It identifies the shortcomings of Hebbian learning in these networks and demonstrates the effectiveness of Contrastive Divergence for training them on image data.

Findings

Hebbian learning causes unwanted behavior and limits network capacity.

Contrastive Divergence improves learning performance.

Networks trained with Contrastive Divergence successfully recognize handwritten digits.

Abstract

Recurrent Bistable Gradient Networks are attractor based neural networks characterized by bistable dynamics of each single neuron. Coupled together using linear interaction determined by the interconnection weights, these networks do not suffer from spurious states or very limited capacity anymore. Vladimir Chinarov and Michael Menzinger, who invented these networks, trained them using Hebb's learning rule. We show, that this way of computing the weights leads to unwanted behaviour and limitations of the networks capabilities. Furthermore we evince, that using the first order of Hintons Contrastive Divergence algorithm leads to a quite promising recurrent neural network. These findings are tested by learning images of the MNIST database for handwritten numbers.