Redundancy Maximization as a Principle of Associative Memory Learning

TL;DR

This paper introduces redundancy maximization as a new principle for designing associative memory systems, significantly increasing their capacity by optimizing local information processing using Partial Information Decomposition.

Contribution

It demonstrates that maximizing redundancy at the neuron level enhances memory capacity, surpassing classical Hopfield networks and establishing a novel information-theoretic design principle.

Findings

Memory capacity increased to 1.59, over ten times the classical Hopfield network.

Redundancy dominates information processing below capacity, with synergy emerging after capacity is exceeded.

Redundancy maximization outperforms recent state-of-the-art associative memory models.

Abstract

Associative memory, traditionally modeled by Hopfield networks, enables the retrieval of previously stored patterns from partial or noisy cues. Yet, the local computational principles which are required to enable this function remain incompletely understood. To formally characterize the local information processing in such systems, we employ a recent extension of information theory - Partial Information Decomposition (PID). PID decomposes the contribution of different inputs to an output into unique information from each input, redundant information across inputs, and synergistic information that emerges from combining different inputs. Applying this framework to individual neurons in classical Hopfield networks we find that below the memory capacity, the information in a neuron's activity is characterized by high redundancy between the external pattern input and the internal recurrent input, while synergy and unique information are close to zero until the memory capacity is surpassed and performance drops steeply. Inspired by this observation, we use redundancy as an information-theoretic learning goal, which is directly optimized for each neuron, dramatically increasing the network's memory capacity to 1.59, a more than tenfold improvement over the 0.14 capacity of classical Hopfield networks and even outperforming recent state-of-the-art implementations of Hopfield networks. Ultimately, this work establishes redundancy maximization as a new design principle for associative memories and opens pathways for new associative memory models based on information-theoretic goals.