Can a Hebbian-like learning rule be avoiding the curse of dimensionality in sparse distributed data?

TL;DR

This paper explores how Hebbian-like learning rules in Restricted Boltzmann Machines can mitigate the curse of dimensionality in sparse data, outperforming traditional backpropagation in generalization with fewer layers.

Contribution

It proposes that Hebbian-like learning in RBMs helps avoid the curse of dimensionality by ignoring zeros, demonstrated through experiments comparing RBMs and backprop-trained networks.

Findings

RBMs generalize well on sparse data

Backprop networks tend to overfit

Hebbian-like learning ignores zeros, reducing dimensionality issues

Abstract

It is generally assumed that the brain uses something akin to sparse distributed representations. These representations, however, are high-dimensional and consequently they affect classification performance of traditional Machine Learning models due to "the curse of dimensionality". In tasks for which there is a vast amount of labeled data, Deep Networks seem to solve this issue with many layers and a non-Hebbian backpropagation algorithm. The brain, however, seems to be able to solve the problem with few layers. In this work, we hypothesize that this happens by using Hebbian learning. Actually, the Hebbian-like learning rule of Restricted Boltzmann Machines learns the input patterns asymmetrically. It exclusively learns the correlation between non-zero values and ignores the zeros, which represent the vast majority of the input dimensionality. By ignoring the zeros "the curse of dimensionality" problem can be avoided. To test our hypothesis, we generated several sparse datasets and compared the performance of a Restricted Boltzmann Machine classifier with some Backprop-trained networks. The experiments using these codes confirm our initial intuition as the Restricted Boltzmann Machine shows a good generalization performance, while the Neural Networks trained with the backpropagation algorithm overfit the training data.