Information Capacity of a Hierarchical Neural Network

TL;DR

This paper analyzes how a hierarchical neural network encodes and categorizes correlated patterns, revealing the relationship between information content, pattern correlation, and network load through analytical and numerical methods.

Contribution

It introduces a measure for non-extensive information content in hierarchical neural networks and characterizes the transition between retrieval and categorization behaviors.

Findings

Transition from retrieval to categorization with increasing examples

Conditions for maximal information based on pattern correlation and load

Numerical simulations validate analytical results

Abstract

The information conveyed by a hierarchical attractor neural network is examined. The network learns sets of correlated patterns (the examples) in the lowest level of the hierarchical tree and can categorize them at the upper levels. A way to measure the non-extensive information content of the examples is formulated. Curves showing the transition from a large retrieval information to a large categorization information behavior, when the number of examples increase, are displayed. The conditions for the maximal information are given as functions of the correlation between examples and the load of concepts. Numerical simulations support the analytical results.