Learning and generalization theories of large committee--machines

TL;DR

This paper derives the critical learning capacity and stability conditions for large committee machines, revealing a Bayesian generalization crossover at a specific capacity related to the number of hidden units.

Contribution

It introduces theoretical formulas for the learning capacity and stability of large committee machines, advancing understanding of their generalization behavior.

Findings

Critical learning capacity $rac{16}{ ext{ extpi}}\, ext{sqrt}( ext{ln} K)$ derived

Stability of solutions verified in the large $K$ limit

Bayesian generalization crossover identified at $ ext{alpha}=K$

Abstract

The study of the distribution of volumes associated to the internal representations of learning examples allows us to derive the critical learning capacity ($\alpha_c=\frac{16}{\pi} \sqrt{\ln K}$) of large committee machines, to verify the stability of the solution in the limit of a large number $K$ of hidden units and to find a Bayesian generalization cross--over at $\alpha=K$.