Training Restricted Boltzmann Machines with Binary Synapses using the Bayesian Learning Rule

TL;DR

This paper introduces a new Bayesian learning rule-based method for training RBMs with binary synapses, eliminating the need for heuristic clipping and improving training efficiency for low-precision neural models.

Contribution

It proposes an alternative optimization approach using natural gradient variational inference, updating natural parameters directly, unlike previous methods that required heuristic clipping.

Findings

No clipping needed due to natural parameter updates

The proposed method is a first-order approximation of Huang's approach

Improves training stability for binary synapse RBMs

Abstract

Restricted Boltzmann machines (RBMs) with low-precision synapses are much appealing with high energy efficiency. However, training RBMs with binary synapses is challenging due to the discrete nature of synapses. Recently Huang proposed one efficient method to train RBMs with binary synapses by using a combination of gradient ascent and the message passing algorithm under the variational inference framework. However, additional heuristic clipping operation is needed. In this technical note, inspired from Huang's work , we propose one alternative optimization method using the Bayesian learning rule, which is one natural gradient variational inference method. As opposed to Huang's method, we update the natural parameters of the variational symmetric Bernoulli distribution rather than the expectation parameters. Since the natural parameters take values in the entire real domain, no additional clipping is needed. Interestingly, the algorithm in \cite{huang2019data} could be viewed as one first-order approximation of the proposed algorithm, which justifies its efficacy with heuristic clipping.