A Max-Sum algorithm for training discrete neural networks

TL;DR

This paper introduces an efficient Max-Sum algorithm for training discrete neural networks, achieving near state-of-the-art complexity and performance, especially for binary and ternary synapses, without approximations.

Contribution

It develops a scalable Max-Sum algorithm for discrete neural network training, outperforming traditional methods in certain settings and handling symmetries in two-layer networks.

Findings

Algorithm scales as O(N log N) per node update.

Performs as well as Belief Propagation on binary perceptrons.

Potentially better suited for fully-connected two-layer networks.

Abstract

We present an efficient learning algorithm for the problem of training neural networks with discrete synapses, a well-known hard (NP-complete) discrete optimization problem. The algorithm is a variant of the so-called Max-Sum (MS) algorithm. In particular, we show how, for bounded integer weights with $q$ distinct states and independent concave a priori distribution (e.g. $l_{1}$ regularization), the algorithm's time complexity can be made to scale as $O\left(N\log N\right)$ per node update, thus putting it on par with alternative schemes, such as Belief Propagation (BP), without resorting to approximations. Two special cases are of particular interest: binary synapses $W\in\{-1,1\}$ and ternary synapses $W\in\{-1,0,1\}$ with $l_{0}$ regularization. The algorithm we present performs as well as BP on binary perceptron learning problems, and may be better suited to address the problem on fully-connected two-layer networks, since inherent symmetries in two layer networks are naturally broken using the MS approach.