Can RBMs be trained with zero step contrastive divergence?

TL;DR

This paper investigates the feasibility of training Restricted Boltzmann Machines (RBMs) using a zero-step contrastive divergence approach, which simplifies the sampling process by eliminating the need for Markov Chain Monte Carlo steps.

Contribution

The paper introduces a modified contrastive divergence method that enables training RBMs with zero MCMC steps, simplifying the training process.

Findings

Zero-step CD can effectively train RBMs on MNIST.

The proposed method reduces computational complexity.

Results show comparable performance to traditional methods.

Abstract

Restricted Boltzmann Machines (RBMs) are probabilistic generative models that can be trained by maximum likelihood in principle, but are usually trained by an approximate algorithm called Contrastive Divergence (CD) in practice. In general, a CD-k algorithm estimates an average with respect to the model distribution using a sample obtained from a k-step Markov Chain Monte Carlo Algorithm (e.g., block Gibbs sampling) starting from some initial configuration. Choices of k typically vary from 1 to 100. This technical report explores if it's possible to leverage a simple approximate sampling algorithm with a modified version of CD in order to train an RBM with k=0. As usual, the method is illustrated on MNIST.