Improving Sparse Associative Memories by Escaping from Bogus Fixed Points

TL;DR

This paper identifies and addresses the issue of bogus fixed points in GBNN associative memories, proposing heuristics and a new post-processing algorithm that significantly improve retrieval accuracy and efficiency.

Contribution

It reveals the existence of bogus fixed points in sum-of-max dynamics and introduces a novel post-processing algorithm to overcome this, enhancing GBNN performance.

Findings

The new algorithm greatly improves retrieval rate.

It reduces the occurrence of bogus fixed points.

Performance in run-time is significantly enhanced.

Abstract

The Gripon-Berrou neural network (GBNN) is a recently invented recurrent neural network embracing a LDPC-like sparse encoding setup which makes it extremely resilient to noise and errors. A natural use of GBNN is as an associative memory. There are two activation rules for the neuron dynamics, namely sum-of-sum and sum-of-max. The latter outperforms the former in terms of retrieval rate by a huge margin. In prior discussions and experiments, it is believed that although sum-of-sum may lead the network to oscillate, sum-of-max always converges to an ensemble of neuron cliques corresponding to previously stored patterns. However, this is not entirely correct. In fact, sum-of-max often converges to bogus fixed points where the ensemble only comprises a small subset of the converged state. By taking advantage of this overlooked fact, we can greatly improve the retrieval rate. We discuss this particular issue and propose a number of heuristics to push sum-of-max beyond these bogus fixed points. To tackle the problem directly and completely, a novel post-processing algorithm is also developed and customized to the structure of GBNN. Experimental results show that the new algorithm achieves a huge performance boost in terms of both retrieval rate and run-time, compared to the standard sum-of-max and all the other heuristics.