Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function

TL;DR

This paper analyzes the expected log partition function of large lattice-gas RBMs using replica theory, revealing differences from Ising RBMs, explicit formulas, stability conditions, and confirming results with simulations.

Contribution

It provides explicit expressions for the lattice-gas RBM log partition function, stability thresholds, and finite size corrections, highlighting differences from Ising RBMs and extending theoretical understanding.

Findings

Explicit formulas for the expected log partition function and variance.

Stability threshold of the replica-symmetric approximation matches message-passing algorithms.

Asymptotic expansions for the log partition function and stability threshold.

Abstract

We study the expectation value of the logarithm of the partition function of large binary-to-binary lattice-gas Restricted Boltzmann Machines (RBMs) within a replica-symmetric ansatz, averaging over the disorder represented by the parameters of the RBM Hamiltonian. Averaging over the Hamiltonian parameters is done with a diagonal covariance matrix. Due to the diagonal form of the parameter covariance matrix not being preserved under the isomorphism between the Ising and lattice-gas forms of the RBM, we find differences in the behaviour of the quenched log partition function of the lattice-gas RBM compared to that of the Ising RBM form usually studied. We obtain explicit expressions for the expectation and variance of the lattice-gas RBM log partition function per node in the thermodynamic limit. We also obtain explicit expressions for the leading order finite size correction to the expected log partition function per node, and the threshold for the stability of the replica-symmetric approximation. We show that the stability threshold of the replica-symmetric approximation is equivalent, in the thermodynamic limit, to the stability threshold of a recent message-passing algorithm used to construct a mean-field Bethe approximation to the RBM free energy. Given the replica-symmetry assumption breaks down as the level of disorder in the spin-spin couplings increases, we obtain asymptotic expansions, in terms of the variance controlling this disorder, for the replica-symmetric log partition function and the replica-symmetric stability threshold. We confirm the various results derived using simulation.