Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning

TL;DR

This paper analyzes dense associative neural networks trained without supervision using statistical mechanics and simulations, revealing their performance and linking physical observables with machine learning loss functions.

Contribution

It provides a comprehensive phase diagram for unsupervised dense neural networks and introduces a novel analytical and computational framework combining Guerra's interpolation and Plefka approximation.

Findings

Identified performance regimes depending on dataset quality and size.

Established a connection between statistical mechanics observables and machine learning loss functions.

Developed a new approach combining large deviations, Guerra's interpolation, and Plefka approximation.

Abstract

We consider dense, associative neural-networks trained with no supervision and we investigate their computational capabilities analytically, via a statistical-mechanics approach, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters such as the quality and quantity of the training dataset and the network storage, valid in the limit of large network size and structureless datasets. Moreover, we establish a bridge between macroscopic observables standardly used in statistical mechanics and loss functions typically used in the machine learning. As technical remarks, from the analytic side, we implement large deviations and stability analysis within Guerra's interpolation to tackle the not-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensors, overall obtaining a novel and broad approach to investigate neural networks in general.