Improved Robustness and Hyperparameter Selection in the Dense Associative Memory

TL;DR

This paper enhances the Dense Associative Memory by addressing computational issues related to large exponents, improving robustness, and enabling more reliable hyperparameter selection without affecting the network's dynamics.

Contribution

The authors identify and mitigate computational problems in Dense Associative Memory, enabling stable implementation and more consistent hyperparameter tuning.

Findings

Modified the network to prevent overflow and imprecision.

Improved hyperparameter selection process.

Maintained network dynamics after modifications.

Abstract

The Dense Associative Memory generalizes the Hopfield network by allowing for sharper interaction functions. This increases the capacity of the network as an autoassociative memory as nearby learned attractors will not interfere with one another. However, the implementation of the network relies on applying large exponents to the dot product of memory vectors and probe vectors. If the dimension of the data is large the calculation can be very large and result in imprecisions and overflow when using floating point numbers in a practical implementation. We describe the computational issues in detail, modify the original network description to mitigate the problem, and show the modification will not alter the networks' dynamics during update or training. We also show our modification greatly improves hyperparameter selection for the Dense Associative Memory, removing dependence on the interaction vertex and resulting in an optimal region of hyperparameters that does not significantly change with the interaction vertex as it does in the original network.