Dense Associative Memories with Analog Circuits

TL;DR

This paper introduces an analog hardware accelerator for Dense Associative Memories (DenseAMs) that performs inference in constant time regardless of model size, leveraging analog circuits for scalable and energy-efficient AI computation.

Contribution

It presents a novel method for building analog DenseAM accelerators using electronic circuits, demonstrating constant-time inference and analyzing scalability and energy efficiency.

Findings

Analog DenseAM hardware achieves constant-time inference.

Inference time scales independently of model size.

Analog technology enables inference in tens to hundreds of nanoseconds.

Abstract

The increasing computational demands of modern AI systems have exposed fundamental limitations of digital hardware, driving interest in alternative paradigms for efficient large-scale inference. Dense Associative Memory (DenseAM) is a family of models that offers a flexible framework for representing many contemporary neural architectures, such as transformers and diffusion models, by casting them as dynamical systems evolving on an energy landscape. In this work, we propose a general method for building analog accelerators for DenseAMs and implementing them using electronic RC circuits, crossbar arrays, and amplifiers. We find that our analog DenseAM hardware performs inference in constant time independent of model size. This result highlights an asymptotic advantage of analog DenseAMs over digital numerical solvers that scale at least linearly with the model size. We consider three settings of progressively increasing complexity: XOR, the Hamming (7,4) code, and a simple language model defined on binary variables. We propose analog implementations of these three models and analyze the scaling of inference time, energy consumption, and hardware. Finally, we estimate lower bounds on the achievable time constants imposed by amplifier specifications, suggesting that even conservative existing analog technology can enable inference times on the order of tens to hundreds of nanoseconds. By harnessing the intrinsic parallelism and continuous-time operation of analog circuits, our DenseAM-based accelerator design offers a new avenue for fast and scalable AI hardware.