Hebbian Memory-Augmented Recurrent Networks: Engram Neurons in Deep Learning

TL;DR

This paper introduces the Engram Neural Network (ENN), a recurrent architecture inspired by biological engrams, incorporating Hebbian plasticity and explicit memory mechanisms to enhance interpretability while maintaining competitive performance.

Contribution

The ENN architecture integrates explicit, differentiable memory with Hebbian plasticity and sparse retrieval, advancing interpretability in recurrent neural networks inspired by neuroscience.

Findings

Achieves comparable accuracy to LSTM and GRU on benchmarks

Provides enhanced interpretability through memory visualization

Demonstrates biologically plausible memory formation processes

Abstract

Despite success across diverse tasks, current artificial recurrent network architectures rely primarily on implicit hidden-state memories, limiting their interpretability and ability to model long-range dependencies. In contrast, biological neural systems employ explicit, associative memory traces (i.e., engrams) strengthened through Hebbian synaptic plasticity and activated sparsely during recall. Motivated by these neurobiological insights, we introduce the Engram Neural Network (ENN), a novel recurrent architecture incorporating an explicit, differentiable memory matrix with Hebbian plasticity and sparse, attention-driven retrieval mechanisms. The ENN explicitly models memory formation and recall through dynamic Hebbian traces, improving transparency and interpretability compared to conventional RNN variants. We evaluate the ENN architecture on three canonical benchmarks: MNIST digit classification, CIFAR-10 image sequence modeling, and WikiText-103 language modeling. Our empirical results demonstrate that the ENN achieves accuracy and generalization performance broadly comparable to classical RNN, GRU, and LSTM architectures, with all models converging to similar accuracy and perplexity on the large-scale WikiText-103 task. At the same time, the ENN offers significant enhancements in interpretability through observable memory dynamics. Hebbian trace visualizations further reveal biologically plausible, structured memory formation processes, validating the potential of neuroscience-inspired mechanisms to inform the development of more interpretable and robust deep learning models.