Theory of the superposition principle for randomized connectionist representations in neural networks

TL;DR

This paper develops a theoretical framework for understanding the superposition principle in neural network representations, analyzing retrieval accuracy, capacity, and proposing models with recency effects to prevent forgetting.

Contribution

It introduces a comprehensive theory mapping superposition operations to linear neural networks, analyzing their capacity and extending models to include recency effects for infinite data streams.

Findings

Networks can achieve about half a bit per neuron in capacity.

Superposition models are mapped to linear neural networks for analysis.

Recency effects help avoid catastrophic forgetting in neural representations.

Abstract

To understand cognitive reasoning in the brain, it has been proposed that symbols and compositions of symbols are represented by activity patterns (vectors) in a large population of neurons. Formal models implementing this idea [Plate 2003], [Kanerva 2009], [Gayler 2003], [Eliasmith 2012] include a reversible superposition operation for representing with a single vector an entire set of symbols or an ordered sequence of symbols. If the representation space is high-dimensional, large sets of symbols can be superposed and individually retrieved. However, crosstalk noise limits the accuracy of retrieval and information capacity. To understand information processing in the brain and to design artificial neural systems for cognitive reasoning, a theory of this superposition operation is essential. Here, such a theory is presented. The superposition operations in different existing models are mapped to linear neural networks with unitary recurrent matrices, in which retrieval accuracy can be analyzed by a single equation. We show that networks representing information in superposition can achieve a channel capacity of about half a bit per neuron, a significant fraction of the total available entropy. Going beyond existing models, superposition operations with recency effects are proposed that avoid catastrophic forgetting when representing the history of infinite data streams. These novel models correspond to recurrent networks with non-unitary matrices or with nonlinear neurons, and can be analyzed and optimized with an extension of our theory.