On associative neural networks for sparse patterns with huge capacities

TL;DR

This paper explores higher-order sparse associative neural networks, demonstrating that increasing interaction order enhances storage capacity from polynomial to super-polynomial scales.

Contribution

It introduces higher-order sparse associative memory models and analyzes their capacity, extending existing models to achieve significantly larger storage capabilities.

Findings

Storage capacity grows polynomially with system size for fixed interaction order.

Allowing interaction order to grow logarithmically yields super-polynomial capacity.

Higher-order interactions improve capacity in sparse associative memory models.

Abstract

Generalized Hopfield models with higher-order or exponential interaction terms are known to have substantially larger storage capacities than the classical quadratic model. On the other hand, associative memories for sparse patterns, such as the Willshaw and Amari models, already outperform the classical Hopfield model in the sparse regime. In this paper we combine these two mechanisms. We introduce higher-order versions of sparse associative memory models and study their storage capacities. For fixed interaction order $n$, we obtain storage capacities of polynomial order in the system size. When the interaction order is allowed to grow logarithmically with the number of neurons, this yields super-polynomial capacities. We also discuss an analogue in the Gripon--Berrou architecture which was formulated for non-sparse messages (see \cite{griponc}). Our results show that the capacity increase caused by higher-order interactions persists in the sparse setting, although the precise storage scale depends on the underlying architecture.