Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network

TL;DR

This paper analytically explores the geometrical structure of weight space in multilayer neural networks, revealing insights into their learning and generalization capabilities, especially for parity and committee machines, with exact results in large networks.

Contribution

It introduces a direct analytical approach to understanding weight space geometry and internal representations in multilayer neural networks, providing new exact results and interpretations.

Findings

Derived the geometrical structure of weight space in MLNs.

Reinterpreted known properties and found new exact results for parity and committee machines.

Established connections with information theory and replica symmetry breaking.

Abstract

We analytically derive the geometrical structure of the weight space in multilayer neural networks (MLN), in terms of the volumes of couplings associated to the internal representations of the training set. Focusing on the parity and committee machines, we deduce their learning and generalization capabilities both reinterpreting some known properties and finding new exact results. The relationship between our approach and information theory as well as the Mitchison--Durbin calculation is established. Our results are exact in the limit of a large number of hidden units, showing that MLN are a class of exactly solvable models with a simple interpretation of replica symmetry breaking.