Learning RBM with a DC programming Approach

TL;DR

This paper introduces a novel stochastic DC programming approach for learning RBMs, which outperforms traditional contrastive divergence in efficiency and effectiveness, especially when using centered gradients.

Contribution

It reformulates RBM training as a difference of convex functions problem, unifies it with contrastive divergence, and improves training efficiency with centered gradients.

Findings

Proposed algorithm reaches higher log-likelihood faster.

Modified algorithm with centered gradients is more efficient.

Outperforms standard contrastive divergence on benchmark datasets.

Abstract

By exploiting the property that the RBM log-likelihood function is the difference of convex functions, we formulate a stochastic variant of the difference of convex functions (DC) programming to minimize the negative log-likelihood. Interestingly, the traditional contrastive divergence algorithm is a special case of the above formulation and the hyperparameters of the two algorithms can be chosen such that the amount of computation per mini-batch is identical. We show that for a given computational budget the proposed algorithm almost always reaches a higher log-likelihood more rapidly, compared to the standard contrastive divergence algorithm. Further, we modify this algorithm to use the centered gradients and show that it is more efficient and effective compared to the standard centered gradient algorithm on benchmark datasets.