Effects of Feature Correlations on Associative Memory Capacity

TL;DR

This paper examines how feature correlations affect the memory capacity of Dense Associative Memory models, revealing that capacity scales exponentially with pattern separation and is slightly reduced by feature correlations, especially at higher polynomial degrees.

Contribution

It introduces an empirical framework to analyze the impact of feature correlations on DAM capacity, bridging theoretical insights with practical data scenarios.

Findings

Memory capacity scales exponentially with pattern separation.

Feature correlations slightly reduce capacity at constant separation.

Higher polynomial degrees amplify the limiting effect of feature correlations.

Abstract

We investigate how feature correlations influence the capacity of Dense Associative Memory (DAM), a Transformer attention-like model. Practical machine learning scenarios involve feature-correlated data and learn representations in the input space, but current capacity analyses do not account for this. We develop an empirical framework to analyze the effects of data structure on capacity dynamics. Specifically, we systematically construct datasets that vary in feature correlation and pattern separation using Hamming distance from information theory, and compute the model's corresponding storage capacity using a simple binary search algorithm. Our experiments confirm that memory capacity scales exponentially with increasing separation in the input space. Feature correlations do not alter this relationship fundamentally, but reduce capacity slightly at constant separation. This effect is amplified at higher polynomial degrees in the energy function, suggesting that Associative Memory is more limited in depicting higher-order interactions between features than patterns. Our findings bridge theoretical work and practical settings for DAM, and might inspire more data-centric methods.