Thermodynamics of Restricted Boltzmann Machines and related learning dynamics

TL;DR

This paper analyzes the thermodynamic properties and learning dynamics of Restricted Boltzmann Machines (RBMs), revealing phase transitions and how data influences the evolution of the model's dominant modes.

Contribution

It provides a thermodynamic framework for RBMs, deriving a phase diagram and modeling the learning process through ensemble averages and order parameters.

Findings

RBMs exhibit a phase diagram similar to the SK model with ferromagnetic phases.

The evolution of singular values is driven by data, with a transition from linear to non-linear regimes.

Experiments confirm RBMs operate in a ferromagnetic compositional phase.

Abstract

We investigate the thermodynamic properties of a Restricted Boltzmann Machine (RBM), a simple energy-based generative model used in the context of unsupervised learning. Assuming the information content of this model to be mainly reflected by the spectral properties of its weight matrix $W$, we try to make a realistic analysis by averaging over an appropriate statistical ensemble of RBMs. First, a phase diagram is derived. Otherwise similar to that of the Sherrington- Kirkpatrick (SK) model with ferromagnetic couplings, the RBM's phase diagram presents a ferromagnetic phase which may or may not be of compositional type depending on the kurtosis of the distribution of the components of the singular vectors of $W$. Subsequently, the learning dynamics of the RBM is studied in the thermodynamic limit. A "typical" learning trajectory is shown to solve an effective dynamical equation, based on the aforementioned ensemble average and explicitly involving order parameters obtained from the thermodynamic analysis. In particular, this let us show how the evolution of the dominant singular values of $W$, and thus of the unstable modes, is driven by the input data. At the beginning of the training, in which the RBM is found to operate in the linear regime, the unstable modes reflect the dominant covariance modes of the data. In the non-linear regime, instead, the selected modes interact and eventually impose a matching of the order parameters to their empirical counterparts estimated from the data. Finally, we illustrate our considerations by performing experiments on both artificial and real data, showing in particular how the RBM operates in the ferromagnetic compositional phase.