Restricted Boltzmann Machine, recent advances and mean-field theory

TL;DR

This review explores the Restricted Boltzmann Machine through the lens of statistical physics, highlighting recent advances in mean-field theory, phase diagrams, and learning algorithms, emphasizing its role in deep learning development.

Contribution

It provides a comprehensive overview of recent theoretical developments in RBMs, especially their analysis via mean-field theory and phase diagrams, connecting statistical physics with machine learning.

Findings

Identification of a compositional phase in RBMs.

Development of mean-field based learning algorithms.

Analysis of ensemble dynamics and stability in RBMs.

Abstract

This review deals with Restricted Boltzmann Machine (RBM) under the light of statistical physics. The RBM is a classical family of Machine learning (ML) models which played a central role in the development of deep learning. Viewing it as a Spin Glass model and exhibiting various links with other models of statistical physics, we gather recent results dealing with mean-field theory in this context. First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM leading in particular to identify a {\it compositional phase} where a small number of features or modes are combined to form complex patterns. Then we discuss recent works either able to devise mean-field based learning algorithms; either able to reproduce generic aspects of the learning process from some {\it ensemble dynamics equations} or/and from linear stability arguments.