Dense Associative Memory with Epanechnikov Energy

TL;DR

This paper introduces a new energy function called log-sum-ReLU (LSR) for Dense Associative Memory networks, inspired by kernel density estimation, which allows for exact memory retrieval with increased local minima and potential for generative applications.

Contribution

The paper proposes the LSR energy function based on the Epanechnikov kernel, enabling exponential memory capacity and emergent local minima in DenseAM networks, a novel approach in the field.

Findings

LSR enables exact memory retrieval with exponential capacity.

LSR introduces many additional local minima with comparable likelihood.

Emergent memories exhibit creativity and potential for generative tasks.

Abstract

We propose a novel energy function for Dense Associative Memory (DenseAM) networks, the log-sum-ReLU (LSR), inspired by optimal kernel density estimation. Unlike the common log-sum-exponential (LSE) function, LSR is based on the Epanechnikov kernel and enables exact memory retrieval with exponential capacity without requiring exponential separation functions. Moreover, it introduces abundant additional \emph{emergent} local minima while preserving perfect pattern recovery -- a characteristic previously unseen in DenseAM literature. Empirical results show that LSR energy has significantly more local minima (memories) that have comparable log-likelihood to LSE-based models. Analysis of LSR's emergent memories on image datasets reveals a degree of creativity and novelty, hinting at this method's potential for both large-scale memory storage and generative tasks.