Modern Hopfield Networks Require Chain-of-Thought to Solve $\mathsf{NC}^1$-Hard Problems

TL;DR

This paper explores the theoretical limits of Modern Hopfield Networks, showing they cannot solve certain complex problems unless augmented with Chain-of-Thought reasoning, which enables solving serial problems beyond their basic capacity.

Contribution

It provides a rigorous complexity-theoretic analysis of MHNs, demonstrating their limitations and showing how Chain-of-Thought mechanisms extend their problem-solving capabilities.

Findings

MHNs are within the TC^0 complexity class and cannot solve NC^1-hard problems.

Adding Chain-of-Thought enables MHNs to solve serial problems like the permutation group word problem.

Limitations of MHNs are characterized, and the role of reasoning steps in overcoming these limits is established.

Abstract

Modern Hopfield Networks (MHNs) have emerged as powerful components in deep learning, serving as effective replacements for pooling layers, LSTMs, and attention mechanisms. While recent advancements have significantly improved their storage capacity and retrieval efficiency, their fundamental theoretical boundaries remain underexplored. In this paper, we rigorously characterize the expressive power of MHNs through the lens of circuit complexity theory. We prove that $\mathrm{poly}(n)$-precision MHNs with constant depth and linear hidden dimension fall within the $\mathsf{DLOGTIME}$-uniform $\mathsf{TC}^0$ complexity class. Consequently, assuming $\mathsf{TC}^0 \neq \mathsf{NC}^1$, we demonstrate that these architectures are incapable of solving $\mathsf{NC}^1$-hard problems, such as undirected graph connectivity and tree isomorphism. We further extend these impossibility results to Kernelized Hopfield Networks. However, we show that these limitations are not absolute: we prove that equipping MHNs with a Chain-of-Thought (CoT) mechanism enables them to transcend the $\mathsf{TC}^0$ barrier, allowing them to solve inherently serial problems like the word problem for the permutation group $S_5$. Collectively, our results delineate a fine-grained boundary between the capabilities of standard MHNs and those augmented with reasoning steps.