Spectral Dynamics of Learning Restricted Boltzmann Machines

TL;DR

This paper analyzes the spectral properties of Restricted Boltzmann Machines (RBMs) during training, revealing how data influences mode selection and providing a deterministic description of the learning dynamics.

Contribution

It introduces a statistical ensemble model for RBM weights and derives equations describing spectral evolution in both linear and non-linear training regimes.

Findings

Spectral properties are driven by data in the initial linear regime.

Unstable modes are selected based on data characteristics.

A deterministic learning curve for RBMs is established.

Abstract

The Restricted Boltzmann Machine (RBM), an important tool used in machine learning in particular for unsupervized learning tasks, is investigated from the perspective of its spectral properties. Starting from empirical observations, we propose a generic statistical ensemble for the weight matrix of the RBM and characterize its mean evolution. This let us show how in the linear regime, in which the RBM is found to operate at the beginning of the training, the statistical properties of the data drive the selection of the unstable modes of the weight matrix. A set of equations characterizing the non-linear regime is then derived, unveiling in some way how the selected modes interact in later stages of the learning procedure and defining a deterministic learning curve for the RBM.