Neural auto-association with optimal Bayesian learning

TL;DR

This paper investigates the optimal Bayesian auto-associative neural network, comparing different learning rules, and introduces an adaptive mechanism that enhances storage capacity, revealing subtle dependencies affecting performance.

Contribution

It introduces an adaptive noise estimation mechanism (ANE) that improves Bayesian learning in auto-associative neural networks, surpassing previous models in storage capacity.

Findings

Performance depends on input pattern dependencies violating naive Bayes assumptions.

Adaptive noise estimation (ANE) significantly enhances storage capacity.

Bayesian learning with ANE achieves maximum storage capacity.

Abstract

Neural associative memories are single layer perceptrons with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Previous works have analyzed the optimal networks under naive Bayes assumptions of independent pattern components and heteroassociation, where the task is to learn associations from input to output patterns. Here I study the optimal Bayesian associative network for auto-association where input and output layers are identical. In particular, I compare performance to different variants of approximate Bayesian learning rules, like the BCPNN (Bayesian Confidence Propagation Neural Network), and try to explain why sometimes the suboptimal learning rules achieve higher storage capacity than the (theoretically) optimal model. It turns out that performance can depend on subtle dependencies of input components violating the ``naive Bayes'' assumptions. This includes patterns with constant number of active units, iterative retrieval where patterns are repeatedly propagated through recurrent networks, and winners-take-all activation of the most probable units. Performance of all learning rules can improve significantly if they include a novel adaptive mechanism to estimate noise in iterative retrieval steps (ANE). The overall maximum storage capacity is achieved again by the Bayesian learning rule with ANE.