Saddle Hierarchy in Dense Associative Memory

TL;DR

This paper analyzes Dense Associative Memory models using statistical mechanics, revealing a hierarchy of saddle points that informs a new regularization scheme, improves training stability, and reduces computational costs.

Contribution

It introduces a saddle hierarchy analysis in DAMs, proposes a novel regularization method, and develops a network-growing algorithm to enhance training efficiency.

Findings

Saddle points in DAMs can be characterized by derived equations.

The new regularization scheme stabilizes training significantly.

The network-growing algorithm reduces training costs drastically.

Abstract

Dense Associative Memory (DAM) models have been attracting renewed attention since they were shown to be robust to adversarial examples and closely related to cutting edge machine learning paradigms, such as the attention mechanism and generative diffusion. We study a DAM built upon a three-layer Boltzmann machine with Potts hidden units, which represent data clusters and classes. Through a statistical mechanics analysis, we derive saddle-point equations that characterize both the stationary points of DAMs trained on real data and the fixed points of DAMs trained on synthetic data within a teacher-student framework. Based on these results, we propose a novel regularization scheme that makes training significantly more stable. Moreover, we show empirically that our DAM learns interpretable solutions to both supervised and unsupervised classification problems. Pushing our theoretical analysis further, we find that the weights learned by relatively small DAMs correspond to unstable saddle points in larger DAMs. We implement a network-growing algorithm that leverages this saddle-point hierarchy to drastically reduce the computational cost of training dense associative memory.