Training of Spiking Neural Networks with Expectation-Propagation

TL;DR

This paper introduces a novel message-passing framework using Expectation-Propagation for training spiking neural networks, enabling gradient-free learning of network parameters and marginalization of nuisance variables, with practical convergence advantages.

Contribution

It presents the first unified, gradient-free training method for both discrete and continuous weights in deterministic and stochastic spiking neural networks.

Findings

Faster convergence than gradient-based methods in practice

Effective training of both discrete and continuous weights

Applicable to deterministic and stochastic spiking networks

Abstract

In this paper, we propose a unifying message-passing framework for training spiking neural networks (SNNs) using Expectation-Propagation. Our gradient-free method is capable of learning the marginal distributions of network parameters and simultaneously marginalizes nuisance parameters, such as the outputs of hidden layers. This framework allows for the first time, training of discrete and continuous weights, for deterministic and stochastic spiking networks, using batches of training samples. Although its convergence is not ensured, the algorithm converges in practice faster than gradient-based methods, without requiring a large number of passes through the training data. The classification and regression results presented pave the way for new efficient training methods for deep Bayesian networks.