A Sparse Quantized Hopfield Network for Online-Continual Memory

TL;DR

This paper introduces the Sparse Quantized Hopfield Network (SQHN), a neural model inspired by brain learning, capable of online, continual memory learning with noisy, non-i.i.d. data, outperforming existing models in various memory tasks.

Contribution

The paper presents the SQHN, a novel neural network model that learns online with local information, mimicking brain-like learning constraints, and demonstrates superior memory performance.

Findings

Outperforms state-of-the-art neural networks on associative memory tasks.

Excels in online, non-i.i.d. learning with noisy inputs.

Shows improved results on a new episodic memory task.

Abstract

An important difference between brains and deep neural networks is the way they learn. Nervous systems learn online where a stream of noisy data points are presented in a non-independent, identically distributed (non-i.i.d.) way. Further, synaptic plasticity in the brain depends only on information local to synapses. Deep networks, on the other hand, typically use non-local learning algorithms and are trained in an offline, non-noisy, i.i.d. setting. Understanding how neural networks learn under the same constraints as the brain is an open problem for neuroscience and neuromorphic computing. A standard approach to this problem has yet to be established. In this paper, we propose that discrete graphical models that learn via an online maximum a posteriori learning algorithm could provide such an approach. We implement this kind of model in a novel neural network called the Sparse Quantized Hopfield Network (SQHN). We show that SQHNs outperform state-of-the-art neural networks on associative memory tasks, outperform these models in online, non-i.i.d. settings, learn efficiently with noisy inputs, and are better than baselines on a novel episodic memory task.