Rectified Lagrangian for Out-of-Distribution Detection in Modern Hopfield Networks

TL;DR

This paper introduces RecLag, a novel Lagrangian for modern Hopfield networks that explicitly models out-of-distribution samples as attractors, significantly improving OOD detection capabilities in high-capacity memory systems.

Contribution

The paper proposes RecLag, a new Lagrangian that incorporates OOD attractors into MHNs, enabling explicit OOD detection and improved performance over existing energy-based methods.

Findings

RecLag enables effective OOD detection in MHNs.

RecLag-based MHNs outperform energy-based OOD detection methods.

The approach is validated across nine image datasets.

Abstract

Modern Hopfield networks (MHNs) have recently gained significant attention in the field of artificial intelligence because they can store and retrieve a large set of patterns with an exponentially large memory capacity. A MHN is generally a dynamical system defined with Lagrangians of memory and feature neurons, where memories associated with in-distribution (ID) samples are represented by attractors in the feature space. One major problem in existing MHNs lies in managing out-of-distribution (OOD) samples because it was originally assumed that all samples are ID samples. To address this, we propose the rectified Lagrangian (RegLag), a new Lagrangian for memory neurons that explicitly incorporates an attractor for OOD samples in the dynamical system of MHNs. RecLag creates a trivial point attractor for any interaction matrix, enabling OOD detection by identifying samples that fall into this attractor as OOD. The interaction matrix is optimized so that the probability densities can be estimated to identify ID/OOD. We demonstrate the effectiveness of RecLag-based MHNs compared to energy-based OOD detection methods, including those using state-of-the-art Hopfield energies, across nine image datasets.