Gaussian-Spherical Restricted Boltzmann Machines

TL;DR

This paper introduces a Gaussian-spherical RBM with a spherical constraint on hidden units, enabling exact asymptotic analysis and revealing spectral dynamics during learning, including mode emergence related to spectral properties.

Contribution

It provides a novel analytical framework for Gaussian-spherical RBMs, linking spectral properties to asymptotic behaviors and learning dynamics, with explicit treatment of finite size effects.

Findings

Exact asymptotic treatments based on spectral properties

Sequential emergence of modes during training

Connection to Bose-Einstein condensation phenomena

Abstract

We consider a special type of Restricted Boltzmann machine (RBM), namely a Gaussian-spherical RBM where the visible units have Gaussian priors while the vector of hidden variables is constrained to stay on an ${\mathbbm L}_2$ sphere. The spherical constraint having the advantage to admit exact asymptotic treatments, various scaling regimes are explicitly identified based solely on the spectral properties of the coupling matrix (also called weight matrix of the RBM). Incidentally these happen to be formally related to similar scaling behaviours obtained in a different context dealing with spatial condensation of zero range processes. More specifically, when the spectrum of the coupling matrix is doubly degenerated an exact treatment can be proposed to deal with finite size effects. Interestingly the known parallel between the ferromagnetic transition of the spherical model and the Bose-Einstein condensation can be made explicit in that case. More importantly this gives us the ability to extract all needed response functions with arbitrary precision for the training algorithm of the RBM. This allows us then to numerically integrate the dynamics of the spectrum of the weight matrix during learning in a precise way. This dynamics reveals in particular a sequential emergence of modes from the Marchenko-Pastur bulk of singular vectors of the coupling matrix.