Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses

TL;DR

This paper uses statistical mechanics to analyze unsupervised feature learning in restricted Boltzmann machines with binary synapses, revealing phase transitions and thermodynamic properties that inform deep network understanding.

Contribution

It introduces a message passing framework and replica theory analysis to characterize phase transitions and thermodynamic behavior in RBMs with binary synapses, advancing theoretical understanding.

Findings

Entropy crisis precedes message passing non-convergence.

Discontinuous phase transition depends on embedded feature strength.

Continuous phase transition observed in certain conditions.

Abstract

Revealing hidden features in unlabeled data is called unsupervised feature learning, which plays an important role in pretraining a deep neural network. Here we provide a statistical mechanics analysis of the unsupervised learning in a restricted Boltzmann machine with binary synapses. A message passing equation to infer the hidden feature is derived, and furthermore, variants of this equation are analyzed. A statistical analysis by replica theory describes the thermodynamic properties of the model. Our analysis confirms an entropy crisis preceding the non-convergence of the message passing equation, suggesting a discontinuous phase transition as a key characteristic of the restricted Boltzmann machine. Continuous phase transition is also confirmed depending on the embedded feature strength in the data. The mean-field result under the replica symmetric assumption agrees with that obtained by running message passing algorithms on single instances of finite sizes. Interestingly, in an approximate Hopfield model, the entropy crisis is absent, and a continuous phase transition is observed instead. We also develop an iterative equation to infer the hyper-parameter (temperature) hidden in the data, which in physics corresponds to iteratively imposing Nishimori condition. Our study provides insights towards understanding the thermodynamic properties of the restricted Boltzmann machine learning, and moreover important theoretical basis to build simplified deep networks.