Annealing Gaussian into ReLU: a New Sampling Strategy for Leaky-ReLU RBM

TL;DR

This paper introduces a novel sampling strategy for leaky RBMs by annealing leakiness instead of temperature, resulting in more efficient likelihood estimation and faster mixing during training.

Contribution

It proposes a new sampling method based on leakiness annealing for leaky RBMs, improving efficiency and accuracy over existing techniques.

Findings

The leakiness annealing method outperforms AIS in likelihood estimation.

The proposed sampler exhibits faster mixing than contrastive divergence.

The approach enhances training efficiency without extra computational cost.

Abstract

Restricted Boltzmann Machine (RBM) is a bipartite graphical model that is used as the building block in energy-based deep generative models. Due to numerical stability and quantifiability of the likelihood, RBM is commonly used with Bernoulli units. Here, we consider an alternative member of exponential family RBM with leaky rectified linear units -- called leaky RBM. We first study the joint and marginal distributions of leaky RBM under different leakiness, which provides us important insights by connecting the leaky RBM model and truncated Gaussian distributions. The connection leads us to a simple yet efficient method for sampling from this model, where the basic idea is to anneal the leakiness rather than the energy; -- i.e., start from a fully Gaussian/Linear unit and gradually decrease the leakiness over iterations. This serves as an alternative to the annealing of the temperature parameter and enables numerical estimation of the likelihood that are more efficient and more accurate than the commonly used annealed importance sampling (AIS). We further demonstrate that the proposed sampling algorithm enjoys faster mixing property than contrastive divergence algorithm, which benefits the training without any additional computational cost.