Shannon Information Capacity of Discrete Synapses

TL;DR

This paper calculates the Shannon information capacity of biological-like discrete synapses, analyzing how factors like synapse number, states, and coding sparseness influence memory storage efficiency.

Contribution

It provides a theoretical framework for understanding the information capacity of discrete synapses, highlighting optimal learning rules and dependencies.

Findings

Storage capacity depends on the number of synapses and states.

Optimal learning rules maximize information storage.

Capacity varies with coding sparseness.

Abstract

There is evidence that biological synapses have only a fixed number of discrete weight states. Memory storage with such synapses behaves quite differently from synapses with unbounded, continuous weights as old memories are automatically overwritten by new memories. We calculate the storage capacity of discrete, bounded synapses in terms of Shannon information. For optimal learning rules, we investigate how information storage depends on the number of synapses, the number of synaptic states and the coding sparseness.