Minimal Perceptrons for Memorizing Complex Patterns

TL;DR

This paper introduces a new complexity measure for binary patterns and predicts the minimal neural network size needed for memorization, validated by simulations, advancing understanding of network design for complex pattern tasks.

Contribution

It develops a novel complexity measure for binary patterns and estimates the minimal network size for memorization based on pattern complexity, including complex patterns.

Findings

Predicted minimal network size matches simulation results.

Complex patterns require larger networks than ordered or disordered ones.

Hamming distance effectively estimates complexity of complex patterns.

Abstract

Feedforward neural networks have been investigated to understand learning and memory, as well as applied to numerous practical problems in pattern classification. It is a rule of thumb that more complex tasks require larger networks. However, the design of optimal network architectures for specific tasks is still an unsolved fundamental problem. In this study, we consider three-layered neural networks for memorizing binary patterns. We developed a new complexity measure of binary patterns, and estimated the minimal network size for memorizing them as a function of their complexity. We formulated the minimal network size for regular, random, and complex patterns. In particular, the minimal size for complex patterns, which are neither ordered nor disordered, was predicted by measuring their Hamming distances from known ordered patterns. Our predictions agreed with simulations based on the back-propagation algorithm.