Neural networks with high order connections

TL;DR

This paper explores high order neuron connections in neural networks, analyzing their equilibrium properties and proposing an optimal learning algorithm that enhances storage capacity and reveals complex phase behaviors.

Contribution

It introduces new high order connection models, analyzes their properties with replica mean-field theory, and presents an improved learning algorithm for fourth order connections.

Findings

Optimal learning algorithm improves storage capacity.

Models exhibit rich phase transition behaviors.

Spin glass states vanish beyond a critical load parameter.

Abstract

We present results for two different kinds of high order connections between neurons acting as corrections to the Hopfield model. Equilibrium properties are analyzed using the replica mean-field theory and compared with numerical simulations. An optimal learning algorithm for fourth order connections is given that improves the storage capacity without increasing the weight of the higher order term. While the behavior of one of the models qualitatively resembles the original Hopfield one, the other presents a new and very rich behavior: depending on the strength of the fourth order connections and the temperature, the system presents two distinct retrieval regions separated by a gap, as well as several phase transitions. Also, the spin glass states seems to disappear above a certain value of the load parameter $\alpha$, $\alpha_g$.