Mean-field message-passing equations in the Hopfield model and its generalizations

TL;DR

This paper revisits mean-field message-passing equations in the Hopfield model, highlighting their use as iterative algorithms and extending them to correlated patterns and layered structures relevant for restricted Boltzmann machines.

Contribution

It explicitly connects belief propagation and TAP equations as message-passing algorithms in the Hopfield model and generalizes TAP equations for correlated patterns and layered structures.

Findings

TAP equations can be used as efficient message-passing algorithms.

Belief propagation depends on the retrieved pattern, unlike TAP.

Modified TAP equations account for correlated patterns and layered structures.

Abstract

Motivated by recent progress in using restricted Boltzmann machines as preprocessing algorithms for deep neural network, we revisit the mean-field equations (belief-propagation and TAP equations) in the best understood such machine, namely the Hopfield model of neural networks, and we explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polarizations of neurons. In the "retrieval phase" where neurons polarize in the direction of one memorized pattern, we point out a major difference between the belief propagation and TAP equations : the set of belief propagation equations depends on the pattern which is retrieved, while one can use a unique set of TAP equations. This makes the latter method much better suited for applications in the learning process of restricted Boltzmann machines. In the case where the patterns memorized in the Hopfield model are not independent, but are correlated through a combinatorial structure, we show that the TAP equations have to be modified. This modification can be seen either as an alteration of the reaction term in TAP equations, or, more interestingly, as the consequence of message passing on a graphical model with several hidden layers, where the number of hidden layers depends on the depth of the correlations in the memorized patterns. This layered structure is actually necessary when one deals with more general restricted Boltzmann machines.