On the role of synaptic stochasticity in training low-precision neural networks

TL;DR

This paper demonstrates that stochastic binary weights in neural networks favor dense, robust solutions with good generalization, contrasting with isolated typical solutions, and introduces a gradient-based method for training such models.

Contribution

It introduces a gradient descent approach for training stochastic binary neural networks, highlighting the importance of synaptic stochasticity in finding robust solutions.

Findings

Stochastic binary weights lead to dense, robust solution regions.

Typical solutions are isolated and hard to find.

The proposed method effectively trains discrete deep neural networks.

Abstract

Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Binary solutions of the standard perceptron problem are obtained from a simple gradient descent procedure on a set of real values parametrizing a probability distribution over the binary synapses. Both analytical and numerical results are presented. An algorithmic extension aimed at training discrete deep neural networks is also investigated.