The emergence of a concept in shallow neural networks

TL;DR

This paper analyzes how shallow neural networks, specifically RBMs, can learn archetypes from blurred data, identifying a critical sample size for successful learning using statistical mechanics and simulations.

Contribution

It introduces a phase diagram for RBMs and Hopfield networks, linking network parameters and data quality to learning success, based on a theoretical and simulation approach.

Findings

Identifies a critical sample size for learning archetypes.

Provides a phase diagram illustrating learning regions.

Uses statistical mechanics and Monte Carlo simulations for analysis.

Abstract

We consider restricted Boltzmann machine (RBMs) trained over an unstructured dataset made of blurred copies of definite but unavailable ``archetypes'' and we show that there exists a critical sample size beyond which the RBM can learn archetypes, namely the machine can successfully play as a generative model or as a classifier, according to the operational routine. In general, assessing a critical sample size (possibly in relation to the quality of the dataset) is still an open problem in machine learning. Here, restricting to the random theory, where shallow networks suffice and the grand-mother cell scenario is correct, we leverage the formal equivalence between RBMs and Hopfield networks, to obtain a phase diagram for both the neural architectures which highlights regions, in the space of the control parameters (i.e., number of archetypes, number of neurons, size and quality of the training set), where learning can be accomplished. Our investigations are led by analytical methods based on the statistical-mechanics of disordered systems and results are further corroborated by extensive Monte Carlo simulations.