Unsupervised Learning by Competing Hidden Units

TL;DR

This paper introduces a biologically plausible, unsupervised learning rule for neural networks that learns useful feature detectors without backpropagation, achieving comparable performance to traditional methods.

Contribution

It proposes a novel unsupervised learning algorithm based on local Hebbian-like rules and global inhibition, enabling feature learning without backpropagation.

Findings

Learned feature detectors are effective for higher layer training.

Performance comparable to standard backpropagation-trained networks.

Method has biological plausibility and unsupervised nature.

Abstract

It is widely believed that the backpropagation algorithm is essential for learning good feature detectors in early layers of artificial neural networks, so that these detectors are useful for the task performed by the higher layers of that neural network. At the same time, the traditional form of backpropagation is biologically implausible. In the present paper we propose an unusual learning rule, which has a degree of biological plausibility, and which is motivated by Hebb's idea that change of the synapse strength should be local - i.e. should depend only on the activities of the pre and post synaptic neurons. We design a learning algorithm that utilizes global inhibition in the hidden layer, and is capable of learning early feature detectors in a completely unsupervised way. These learned lower layer feature detectors can be used to train higher layer weights in a usual supervised way so that the performance of the full network is comparable to the performance of standard feedforward networks trained end-to-end with a backpropagation algorithm.