Analytical and Numerical Study of Internal Representations in Multilayer Neural Networks with Binary Weights

TL;DR

This paper investigates the internal representations of multilayer neural networks with binary weights, analyzing their structure, learning behavior, and symmetry breaking transitions through analytical and numerical methods.

Contribution

It provides a detailed analysis of the weight space structure and internal representations in binary-weight neural networks, including the distribution of volumes and phase transitions.

Findings

Distribution of internal representation volumes derived

Learning behavior and symmetry breaking transition characterized

Results validated with numerical simulations

Abstract

We study the weight space structure of the parity machine with binary weights by deriving the distribution of volumes associated to the internal representations of the learning examples. The learning behaviour and the symmetry breaking transition are analyzed and the results are found to be in very good agreement with extended numerical simulations.