Variational Probability Flow for Biologically Plausible Training of Deep Neural Networks

TL;DR

This paper introduces Variational Probability Flow (VPF), a biologically plausible training algorithm for deep neural networks that improves efficiency and local learning rules, demonstrated on MNIST datasets with promising results.

Contribution

The paper presents VPF, a novel training method for deep Boltzmann Machines that is local, does not require Gibbs sampling or backpropagation, and can handle intra-layer connections.

Findings

VPF learns features quickly on MNIST datasets.

VPF reconstructs corrupted images more accurately.

VPF generates high-likelihood samples.

Abstract

The quest for biologically plausible deep learning is driven, not just by the desire to explain experimentally-observed properties of biological neural networks, but also by the hope of discovering more efficient methods for training artificial networks. In this paper, we propose a new algorithm named Variational Probably Flow (VPF), an extension of minimum probability flow for training binary Deep Boltzmann Machines (DBMs). We show that weight updates in VPF are local, depending only on the states and firing rates of the adjacent neurons. Unlike contrastive divergence, there is no need for Gibbs confabulations; and unlike backpropagation, alternating feedforward and feedback phases are not required. Moreover, the learning algorithm is effective for training DBMs with intra-layer connections between the hidden nodes. Experiments with MNIST and Fashion MNIST demonstrate that VPF learns reasonable features quickly, reconstructs corrupted images more accurately, and generates samples with a high estimated log-likelihood. Lastly, we note that, interestingly, if an asymmetric version of VPF exists, the weight updates directly explain experimental results in Spike-Timing-Dependent Plasticity (STDP).