On Unstable Fixed Points in Modern Continuous Hopfield Networks

TL;DR

This paper analyzes the fixed point structure of continuous Hopfield networks, showing that under certain geometric conditions, unstable fixed points necessarily exist, extending previous theoretical results.

Contribution

It demonstrates that natural geometric conditions in continuous Hopfield networks imply the existence of unstable fixed points, complementing prior work on their structure.

Findings

Unstable fixed points occur under natural geometric conditions.

Such fixed points are associated with higher-dimensional faces of the pattern polytope.

Results extend understanding of fixed point stability in continuous Hopfield networks.

Abstract

The recently introduced continuous Hopfield network (see Ramsauer et al.) exhibits large memorization capabilities, which manifest as attractive fixed points of its update rule -- a differentiable function consisting of two linear mappings composed with the scaled softmax function. The authors of the aforementioned work provide proofs for the existence and approximate position of such attractive fixed points. For the softmax function alone, the fixed point structure has been fully characterized in earlier work by P. Ti\v{n}o, from which it turns out that for sufficiently large scaling factors there are exponentially more unstable fixed points than attractive ones. In this work, we complement the findings of Ramsauer et al. by showing that, under natural geometric conditions on the vectors defining the continuous Hopfield network, unstable fixed points must occur, analogous to the findings of Ti\v{n}o. Our results show that, under these geometric conditions, continuous Hopfield networks necessarily admit additional unstable fixed points associated with higher-dimensional faces of the pattern polytope.