Storage capacity of networks with discrete synapses and sparsely encoded memories

TL;DR

This paper analyzes the storage capacity of neural networks with discrete synapses and sparse coding, showing they approach optimal capacity and are more efficient per synapse than fully connected networks.

Contribution

It demonstrates that networks with discrete synapses and sparse coding can nearly reach optimal storage capacity and are more efficient than fully connected networks.

Findings

Discrete synapses have similar capacity to continuous ones.

Capacity approaches optimal in sparse coding limit.

Sparse binary networks store information more efficiently.

Abstract

Attractor neural networks (ANNs) are one of the leading theoretical frameworks for the formation and retrieval of memories in networks of biological neurons. In this framework, a pattern imposed by external inputs to the network is said to be learned when this pattern becomes a fixed point attractor of the network dynamics. The storage capacity is the maximum number of patterns that can be learned by the network. In this paper, we study the storage capacity of fully-connected and sparsely-connected networks with a binarized Hebbian rule, for arbitrary coding levels. Our results show that a network with discrete synapses has a similar storage capacity as the model with continuous synapses, and that this capacity tends asymptotically towards the optimal capacity, in the space of all possible binary connectivity matrices, in the sparse coding limit. We also derive finite coding level corrections for the asymptotic solution in the sparse coding limit. The result indicates the capacity of network with Hebbian learning rules converges to the optimal capacity extremely slowly when the coding level becomes small. Our results also show that in networks with sparse binary connectivity matrices, the information capacity per synapse is larger than in the fully connected case, and thus such networks store information more efficiently.