A Neighbourhood-Based Stopping Criterion for Contrastive Divergence Learning

TL;DR

This paper proposes a neighborhood-based stopping criterion for Contrastive Divergence learning in RBMs, addressing the limitations of using reconstruction error alone by incorporating information from neighboring states.

Contribution

It introduces a novel stopping criterion leveraging neighboring states to improve the training process of RBMs with Contrastive Divergence.

Findings

Neighborhood-based criterion better estimates optimal stopping point

Reduces overfitting compared to reconstruction error

Improves convergence stability in RBM training

Abstract

Restricted Boltzmann Machines (RBMs) are general unsupervised learning devices to ascertain generative models of data distributions. RBMs are often trained using the Contrastive Divergence learning algorithm (CD), an approximation to the gradient of the data log-likelihood. A simple reconstruction error is often used as a stopping criterion for CD, although several authors \cite{schulz-et-al-Convergence-Contrastive-Divergence-2010-NIPSw, fischer-igel-Divergence-Contrastive-Divergence-2010-ICANN} have raised doubts concerning the feasibility of this procedure. In many cases the evolution curve of the reconstruction error is monotonic while the log-likelihood is not, thus indicating that the former is not a good estimator of the optimal stopping point for learning. However, not many alternatives to the reconstruction error have been discussed in the literature. In this manuscript we investigate simple alternatives to the reconstruction error, based on the inclusion of information contained in neighboring states to the training set, as a stopping criterion for CD learning.