Generalization learning in a perceptron with binary synapses

TL;DR

This paper analyzes the generalization capabilities of a perceptron with binary synapses using the SBPI learning algorithm, providing a mean-field theoretical framework and demonstrating its efficiency over traditional algorithms.

Contribution

The paper introduces a mean-field analysis of SBPI in binary perceptrons, revealing its convergence properties and relation to a simpler stochastic meta-plastic reinforcement algorithm.

Findings

SBPI converges in order N*sqrt(log(N)) time.

Clipped perceptron does not converge in the considered timeframe.

SBPI is equivalent to a simpler stochastic reinforcement algorithm.

Abstract

We consider the generalization problem for a perceptron with binary synapses, implementing the Stochastic Belief-Propagation-Inspired (SBPI) learning algorithm which we proposed earlier, and perform a mean-field calculation to obtain a differential equation which describes the behaviour of the device in the limit of a large number of synapses N. We show that the solving time of SBPI is of order N*sqrt(log(N)), while the similar, well-known clipped perceptron (CP) algorithm does not converge to a solution at all in the time frame we considered. The analysis gives some insight into the ongoing process and shows that, in this context, the SBPI algorithm is equivalent to a new, simpler algorithm, which only differs from the CP algorithm by the addition of a stochastic, unsupervised meta-plastic reinforcement process, whose rate of application must be less than sqrt(2/(\pi * N)) for the learning to be achieved effectively. The analytical results are confirmed by simulations.