Kernel Memory Networks: A Unifying Framework for Memory Modeling

TL;DR

This paper introduces kernel memory networks, a unifying framework for memory modeling in neural networks that enhances noise robustness and storage capacity, encompassing many existing memory models and offering new biological insights.

Contribution

The paper develops a novel kernel memory network framework that unifies and extends previous memory models, enabling storage of exponential patterns with biological plausibility.

Findings

Can store exponential number of patterns

Includes many existing memory models as special cases

Provides biological interpretations of memory mechanisms

Abstract

We consider the problem of training a neural network to store a set of patterns with maximal noise robustness. A solution, in terms of optimal weights and state update rules, is derived by training each individual neuron to perform either kernel classification or interpolation with a minimum weight norm. By applying this method to feed-forward and recurrent networks, we derive optimal models, termed kernel memory networks, that include, as special cases, many of the hetero- and auto-associative memory models that have been proposed over the past years, such as modern Hopfield networks and Kanerva's sparse distributed memory. We modify Kanerva's model and demonstrate a simple way to design a kernel memory network that can store an exponential number of continuous-valued patterns with a finite basin of attraction. The framework of kernel memory networks offers a simple and intuitive way to understand the storage capacity of previous memory models, and allows for new biological interpretations in terms of dendritic non-linearities and synaptic cross-talk.