Do Hopfield Networks Dream of Stored Patterns? A Statistical-Mechanical Theory of Dreaming in Multidirectional Associative Memories

TL;DR

This paper introduces a multi-layer Hebbian architecture called DLAM that models off-line dreaming within energy-based neural networks, enhancing pattern retrieval and disentanglement through a synergy of dreaming and inter-layer coupling.

Contribution

It develops a novel theoretical framework for energy-based models incorporating dreaming and heteroassociative coupling, extending Hopfield networks' capabilities for complex neural tasks.

Findings

Dreaming attenuates interference modes, reducing crosstalk.

Dreaming and coupling work together to access new retrieval regions.

Off-line consolidation can substitute for more training data.

Abstract

We introduce the Dreaming $L$-directional Associative Memory (DLAM), a multi-layer Hebbian architecture in which off-line dreaming and supervised heteroassociative coupling coexist within a single energy function, placing our approach within the framework of energy-based models (EBMs). The replica-symmetric free energy, derived via the Guerra interpolation scheme, yields self-consistency equations governing the order parameters across the control-parameter space. The effective local field decomposes into signal, intra-layer dreaming noise, and inter-layer noise. Dreaming improves retrieval by differentially attenuating high-eigenvalue interference modes of the empirical correlation matrix, suppressing inter-pattern crosstalk while preserving the signal. Dreaming and inter-layer coupling prove synergistic, opening retrieval regions unreachable by either mechanism alone, as confirmed by Monte Carlo simulations for $L=3$. Their interplay is most pronounced on pattern disentanglement: given a mixture state as input, the network splits the constituent patterns one-per-layer, recovering each modality-specific pattern from a common cue that simultaneously blends noisy evidence from all sensory channels. Phase diagrams are planar projections of the hyperspace $(\alpha,\beta,\rho,t)$-where $\alpha$ is the storage load, $\beta$ the fast-noise inverse temperature, $\rho$ the dataset entropy, and $t$ the sleeping time. In the $(\rho,t)$-plane, the diagrams reveal a data-computation trade-off: off-line consolidation substitutes for additional training data, extending to heteroassociative architectures a phenomenon previously established for autoassociative networks. Enriching the standard Hopfield model with heteroassociativity and dreaming gives rise to EBMs capable of complex tasks beyond classical pattern recognition, contributing to a modern theory of neural information processing.