Benchmarking Hebbian learning rules for associative memory

TL;DR

This paper benchmarks six Hebbian learning rules for associative memory, focusing on storage capacity and prototype extraction, and finds BCPNN generally outperforms other rules across various conditions.

Contribution

It provides a comprehensive comparison of Hebbian learning rules for associative memory, emphasizing prototype extraction and evaluating different network architectures and data sets.

Findings

BCPNN outperforms other rules in most cases

Covariance learning has low storage capacity

Performance varies with network architecture and data correlation

Abstract

Associative memory or content addressable memory is an important component function in computer science and information processing and is a key concept in cognitive and computational brain science. Many different neural network architectures and learning rules have been proposed to model associative memory of the brain while investigating key functions like pattern completion and rivalry, noise reduction, and storage capacity. A less investigated but important function is prototype extraction where the training set comprises pattern instances generated by distorting prototype patterns and the task of the trained network is to recall the correct prototype pattern given a new instance. In this paper we characterize these different aspects of associative memory performance and benchmark six different learning rules on storage capacity and prototype extraction. We consider only models with Hebbian plasticity that operate on sparse distributed representations with unit activities in the interval [0,1]. We evaluate both non-modular and modular network architectures and compare performance when trained and tested on different kinds of sparse random binary pattern sets, including correlated ones. We show that covariance learning has a robust but low storage capacity under these conditions and that the Bayesian Confidence Propagation learning rule (BCPNN) is superior with a good margin in all cases except one, reaching a three times higher composite score than the second best learning rule tested.