Stopping Criteria in Contrastive Divergence: Alternatives to the Reconstruction Error

TL;DR

This paper explores alternative stopping criteria to the reconstruction error for training Restricted Boltzmann Machines with Contrastive Divergence, aiming to better detect improvements in the model's log-likelihood during learning.

Contribution

It introduces and evaluates simple alternative metrics to the reconstruction error for more effective stopping decisions in RBM training.

Findings

Alternative criteria can better indicate log-likelihood improvements

Reconstruction error may not reliably signal training progress

Proposed methods improve training efficiency

Abstract

Restricted Boltzmann Machines (RBMs) are general unsupervised learning devices to ascertain generative models of data distributions. RBMs are often trained using the Contrastive Divergence learning algorithm (CD), an approximation to the gradient of the data log-likelihood. A simple reconstruction error is often used to decide whether the approximation provided by the CD algorithm is good enough, though several authors (Schulz et al., 2010; Fischer & Igel, 2010) have raised doubts concerning the feasibility of this procedure. However, not many alternatives to the reconstruction error have been used in the literature. In this manuscript we investigate simple alternatives to the reconstruction error in order to detect as soon as possible the decrease in the log-likelihood during learning.