Efficiency of local learning rules in threshold-linear associative networks

TL;DR

This paper analyzes the storage capacity of threshold-linear associative networks, demonstrating that Hebbian learning can approach or surpass theoretical bounds more efficiently than binary networks, primarily through pattern sparsification.

Contribution

It derives the Gardner capacity for threshold-linear units and shows Hebbian learning's efficiency in approaching optimal storage bounds.

Findings

Hebbian learning allows networks to operate near or above Gardner capacity.

Pattern sparsification enhances storage efficiency.

Hebbian learning is a neurally plausible alternative to backpropagation.

Abstract

We derive the Gardner storage capacity for associative networks of threshold linear units, and show that with Hebbian learning they can operate closer to such Gardner bound than binary networks, and even surpass it. This is largely achieved through a sparsification of the retrieved patterns, which we analyze for theoretical and empirical distributions of activity. As reaching the optimal capacity via non-local learning rules like backpropagation requires slow and neurally implausible training procedures, our results indicate that one-shot self-organized Hebbian learning can be just as efficient.