Dense Hebbian neural networks: a replica symmetric picture of supervised learning

TL;DR

This paper analyzes dense associative neural networks trained with supervision, using statistical mechanics and simulations, revealing their capacity for ultra-storage and ultra-detection regimes, and testing their performance on structured datasets like MNist.

Contribution

It introduces a comprehensive analytical and numerical framework for understanding supervised learning in dense neural networks beyond shallow models.

Findings

Networks can operate in ultra-storage regime handling many patterns.

Networks can perform pattern recognition at high noise levels.

The framework applies to structured datasets like MNist.

Abstract

We consider dense, associative neural-networks trained by a teacher (i.e., with supervision) and we investigate their computational capabilities analytically, via statistical-mechanics of spin glasses, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters such as quality and quantity of the training dataset, network storage and noise, that is valid in the limit of large network size and structureless datasets: these networks may work in a ultra-storage regime (where they can handle a huge amount of patterns, if compared with shallow neural networks) or in a ultra-detection regime (where they can perform pattern recognition at prohibitive signal-to-noise ratios, if compared with shallow neural networks). Guided by the random theory as a reference framework, we also test numerically learning, storing and retrieval capabilities shown by these networks on structured datasets as MNist and Fashion MNist. As technical remarks, from the analytic side, we implement large deviations and stability analysis within Guerra's interpolation to tackle the not-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensors, overall obtaining a novel and broad approach to investigate supervised learning in neural networks, beyond the shallow limit, in general.