Three Learning Stages and Accuracy-Efficiency Tradeoff of Restricted Boltzmann Machines

TL;DR

This paper investigates the learning process of Restricted Boltzmann Machines, identifying three distinct regimes that balance accuracy and efficiency, and highlighting the inherent tradeoffs involved.

Contribution

It introduces a framework to understand the three learning regimes of RBMs and quantifies the tradeoff between accuracy and efficiency during training.

Findings

Identifies three regimes: independent learning, correlation learning, degradation.

Quantifies the tradeoff between accuracy and efficiency in RBM training.

Provides numerical evidence and heuristic analysis of the regimes.

Abstract

Restricted Boltzmann Machines (RBMs) offer a versatile architecture for unsupervised machine learning that can in principle approximate any target probability distribution with arbitrary accuracy. However, the RBM model is usually not directly accessible due to its computational complexity, and Markov-chain sampling is invoked to analyze the learned probability distribution. For training and eventual applications, it is thus desirable to have a sampler that is both accurate and efficient. We highlight that these two goals generally compete with each other and cannot be achieved simultaneously. More specifically, we identify and quantitatively characterize three regimes of RBM learning: independent learning, where the accuracy improves without losing efficiency; correlation learning, where higher accuracy entails lower efficiency; and degradation, where both accuracy and efficiency no longer improve or even deteriorate. These findings are based on numerical experiments and heuristic arguments.