A mathematical analysis of hierarchical Hopfield models

TL;DR

This paper provides a rigorous mathematical analysis of hierarchical Hopfield models, focusing on how structured information like features, strokes, and concepts can be stored and retrieved, even with noisy data.

Contribution

It introduces a formalism for structuring information in hierarchical Hopfield models and derives criteria for successful concept retrieval despite errors at the stroke level.

Findings

Concepts can be retrieved from noisy input data.

Second-layer retrieval compensates for first-layer errors.

Criteria for successful retrieval depend on concept size and noise level.

Abstract

The central question that we address is: How can structured information be stored in a hierarchical Hopfield model involving hidden layers? To this end, we develop a formalism of strokes and concepts that allows us to appropriately structure information: initial features are first classified into strokes, which in a second step are aggregated into concepts. We rigorously derive criteria under which concepts can be retrieved from noisy input data. A remarkable effect is that we do not require a perfect retrieval at the level of strokes, as the second-layer retrieval procedure compensates for first-layer errors. We treat separately the cases of fixed and variable-sized concepts.