Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions

TL;DR

This paper analyzes the training dynamics of Restricted Boltzmann Machines in high dimensions using advanced mathematical tools, revealing their optimal recovery capabilities in structured data models.

Contribution

It introduces a novel asymptotic analysis of RBM training using AMP and mean-field theory, connecting RBM performance to the BBP transition in high-dimensional settings.

Findings

RBM reaches the optimal weak recovery threshold in the spiked covariance model.

The analysis connects RBM training to multi-index models and established high-dimensional methods.

The results provide rigorous insights into RBM performance in structured data regimes.

Abstract

The Restricted Boltzmann Machine (RBM) is one of the simplest generative neural networks capable of learning input distributions. Despite its simplicity, the analysis of its performance in learning from the training data is only well understood in cases that essentially reduce to singular value decomposition of the data. Here, we consider the limit of a large dimension of the input space and a constant number of hidden units. In this limit, we simplify the standard RBM training objective into a form that is equivalent to the multi-index model with non-separable regularization. This opens a path to analyze training of the RBM using methods that are established for multi-index models, such as Approximate Message Passing (AMP) and its state evolution, and the analysis of Gradient Descent (GD) via the dynamical mean-field theory. We then give rigorous asymptotics of the training dynamics of RBM on data generated by the spiked covariance model as a prototype of a structure suitable for unsupervised learning. We show in particular that RBM reaches the optimal computational weak recovery threshold, aligning with the BBP transition, in the spiked covariance model.