iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixer

TL;DR

This paper introduces iMixer, a hierarchical Hopfield network-based model that generalizes MLP-Mixer, demonstrating invertible, implicit, and iterative properties, and achieves competitive image classification performance, providing insights into Transformer-like architectures.

Contribution

The paper proposes iMixer, a novel hierarchical Hopfield network-based model that generalizes MLP-Mixer with invertible and iterative features, advancing understanding of Transformer-like architectures.

Findings

iMixer achieves performance comparable or superior to vanilla MLP-Mixer.

iMixer exhibits stable learning despite its invertible, implicit, and iterative structure.

The study supports the theoretical link between Hopfield networks and Transformer architectures.

Abstract

In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models that compete with Transformers, such as the MLP-Mixer. Despite their weak inductive bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures design. In this paper, we generalize the correspondence to the recently introduced hierarchical Hopfield network, and find iMixer, a novel generalization of MLP-Mixer model. Unlike ordinary feedforward neural networks, iMixer involves MLP layers that propagate forward from the output side to the input side. We characterize the module as an example of invertible, implicit, and iterative mixing module. We evaluate the model performance with various datasets on image classification tasks, and find that iMixer, despite its unique architecture, exhibits stable learning capabilities and achieves performance comparable to or better than the baseline vanilla MLP-Mixer. The results imply that the correspondence between the Hopfield networks and the Mixer models serves as a principle for understanding a broader class of Transformer-like architecture designs.