A phase transition in sampling from Restricted Boltzmann Machines

TL;DR

This paper reveals a phase transition in the mixing time of Gibbs sampling for Restricted Boltzmann Machines, showing how it changes dramatically depending on a key parameter, with implications for understanding their computational complexity.

Contribution

We establish a phase transition in the mixing time of Gibbs sampling in RBMs and connect it to a dynamical system analysis, introducing a new isoperimetric inequality for the critical case.

Findings

Mixing time varies logarithmically, polynomially, and exponentially with system size depending on the parameter.

A critical parameter value c* approximately -5.87 marks the transition point.

The stationary distribution is shown to be nearly log-concave at the critical point.

Abstract

Restricted Boltzmann Machines are a class of undirected graphical models that play a key role in deep learning and unsupervised learning. In this study, we prove a phase transition phenomenon in the mixing time of the Gibbs sampler for a one-parameter Restricted Boltzmann Machine. Specifically, the mixing time varies logarithmically, polynomially, and exponentially with the number of vertices depending on whether the parameter $c$ is above, equal to, or below a critical value $c_\star\approx-5.87$. A key insight from our analysis is the link between the Gibbs sampler and a dynamical system, which we utilize to quantify the former based on the behavior of the latter. To study the critical case $c= c_\star$, we develop a new isoperimetric inequality for the sampler's stationary distribution by showing that the distribution is nearly log-concave.