Advanced Mean Field Theory of Restricted Boltzmann Machine

TL;DR

This paper introduces an advanced mean field theory for restricted Boltzmann machines that enables efficient computation of network statistics and gradients without sampling, improving over traditional methods.

Contribution

It develops a novel mean field approach based on the Bethe approximation for RBMs, eliminating the need for sampling in gradient evaluation.

Findings

Accurately estimates partition function and free energy

Provides efficient message passing algorithm

Matches results of sampling-based methods

Abstract

Learning in restricted Boltzmann machine is typically hard due to the computation of gradients of log-likelihood function. To describe the network state statistics of the restricted Boltzmann machine, we develop an advanced mean field theory based on the Bethe approximation. Our theory provides an efficient message passing based method that evaluates not only the partition function (free energy) but also its gradients without requiring statistical sampling. The results are compared with those obtained by the computationally expensive sampling based method.